Freeradius 3.0.12 problems - redis and mysql pool connections
Hello Alan, I found 2 problems in freeradius 3.0.12 To be short: Our several freeradius servers are using MySQL cluster (percona) and redis master/slave mapped to local port using HA Proxy on each freeradius server. So in fact each Freeradius sees 1 local port for MySQL and 1 local port for redis. We are not using server pool and load balancing definition in Freeradius in order to have strict control on which database server is used for all writes from several Freeradius servers. To be sure Freeradius is starting when redis or sql is not started yet (or not available for a while) in both redis and sql pool definition we have to use: pool { start = 0 but we also want to have min number of connections defined higher than 0 (e.g. in a standard way): min = ${thread[pool].min_spare_servers} So min > start. When server (sql or redis) becomes available Freeradius opens some connections. After restart of sql or redis Freeradius should open new connections to those servers as old connections are not longer available. And we have 2 errors #1 When redis becomes unavailable for a while (restart of redis-server or simple restart of HA_Proxy) Freeradius does not check that connection is no longer available and keeps those invalid connections till lifetime time. In fact in such situation old connection should be discarded and new connection shall be opened. Standard lifetime in mods-available/redis is set to 86400 so redis module is not able to read/write anything from redis-server for 24 hours. In /var/log/freeradius/radius.log there are only ERRORs when there is redis write command, there is no error information when there is (unsuccessful) read command. To see that debug mode shall be started. So in fact redis module is not verifying redis valid/invalid connection state. Changing lifetime to e.g. 60 helps to force redis module start working (in 60 second from redis-server restart) but that is just a workaround not a solution. #2 In var/log/freeradius/radius.log we have lots of errors: ERROR: SQL query failed: no connection It seems that error shows up when connection is closed by e.g. idle-timeout... Example: Thu Dec 22 15:48:35 2016 : Debug: attribute --> User-Name Thu Dec 22 15:48:35 2016 : Debug: (81) EXPAND %{User-Name} Thu Dec 22 15:48:35 2016 : Debug: (81) --> h274_xxxxxxxxxxxxxx Thu Dec 22 15:48:35 2016 : Debug: (81) SQL-User-Name set to 'h274_xxxxxxxxxxxxxx' Thu Dec 22 15:48:35 2016 : Debug: rlm_sql (sql): Reserved connection (4) Thu Dec 22 15:48:35 2016 : Debug: (81) Executing select query: INSERT INTO radius_radacct_closed ( acctsessionid, acctuniqueid, username, hotspot_id, device_id, felogin_id, realm, nasipaddress, nasportid, nasporttype, acctstarttime, acctupdatetime, acctstoptime, acctsessiontime, acctauthentic, connectinfo_start, connectinfo_stop, acctinputoctets, acctoutputoctets, calledstationid, callingstationid, acctterminatecause, servicetype, framedprotocol, framedipaddress ) VALUES ('585be1cc00000004', 'e2a5c5294e55743ffc9169de47d5de4a', 'h274_xxxxxxxxxxxxxx', '274', '3625', '3799789', '', '192.168.176.1', '00000004', 'Wireless-802.11', FROM_UNIXTIME(1482418115 - 1460), FROM_UNIXTIME(1482418115), FROM_UNIXTIME(1482418115), 1460, '', '', '', '0' << 32 | '129433', '0' << 32 | '1435982', 'h274-xxxxxxxxxxxxxx', 'EC-9B-F3-19-4A-F2', 'Lost-Carrier', '', '', '192.168.176.5') Thu Dec 22 15:48:35 2016 : ERROR: (81) SQL query failed: no connection Thu Dec 22 15:48:35 2016 : Debug: rlm_sql (sql): Released connection (4) Thu Dec 22 15:48:35 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:48:35 2016 : Debug: (81) EXPAND %{sql: INSERT INTO radius_radacct_closed ( acctsessionid, acctuniqueid, username, hotspot_id, device_id, felogin_id, realm, nasipaddress, nasportid, nasporttype, acctstarttime, acctupdatetime, acctstoptime, acctsessiontime, acctauthentic, connectinfo_start, connectinfo_stop, acctinputoctets, acctoutputoctets, calledstationid, callingstationid, acctterminatecause, servicetype, framedprotocol, framedipaddress ) VALUES ('%{Acct-Session-Id}', '%{Acct-Unique-Session-Id}', '%{User-Name}', '%{LOCATION-HotspotID}', '%{LOCATION-DeviceID}', '%{LOCATION-FELogin-Id}', '%{Realm}', '%{NAS-IP-Address}', '%{%{NAS-Port-ID}:-%{NAS-Port}}', '%{NAS-Port-Type}', FROM_UNIXTIME(%l - %{%{Acct-Session-Time}:-0}), FROM_UNIXTIME(%l), FROM_UNIXTIME(%l), %{%{Acct-Session-Time}:-NULL}, '%{Acct-Authentic}', '', '%{Connect-Info}', '%{%{Acct-Input-Gigawords}:-0}' << 32 | '%{%{Acct-Input-Octets}:-0}', '%{%{Acct-Output-Gigawords}:-0}' << 32 | '%{%{Acct-Output-Octets}:-0}', '%{Called-Station-Id}', '%{Calling-Station-Id}', '%{Acct-Terminate-Cause}', '%{Service-Type}', '%{Framed-Protocol}', '%{Framed-IP-Address}') } Thu Dec 22 15:48:35 2016 : Debug: (81) --> freeradius log looks like: radius1:~$ tail /var/log/freeradius/radius.log Thu Dec 22 15:59:45 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:47 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:47 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:48 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:48 2016 : ERROR: (164) ERROR: SQL query failed: no connection Thu Dec 22 15:59:48 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:51 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : ERROR: (172) ERROR: SQL query failed: no connection We tested to set freeradius use mysql directly, without haproxy and using default values in mods-availables/sql (including pool { start value) but it did not change this strange behavior. Any help would be appreciated... Regards, Mike ________________________________________ Uwaga: Treść niniejszej wiadomości może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. ________________________________ Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.If you have received this communication in error, please notify the sender immediately by replying to the message and deleting it from your computer. Thank you. ________________________________
On 22/12/2016 15:30, Michal Tomaszewski wrote:
To be sure Freeradius is starting when redis or sql is not started yet (or not available for a while) in both redis and sql pool definition we have to use: pool { start = 0
If you are on a systemd-based system (e.g. CentOS 7, Ubuntu 16.04), another option is to get systemd to restart freeradius for you. systemctl edit radiusd # or systemctl edit freeradius This will throw you into an editor. Type: [Service] Restart=always RestartSec=3 and exit and save. (Note: exact capitalisation is important) Then if freeradius fails to start (e.g. because it can't open a connection to a backend), systemd will restart it every 3 seconds. In addition, if freeradius dies for any reason, it will be restarted. Adjust RestartSec as required - the default is 0.1s which is too aggressive for my taste. Regards, Brian.
On Dec 22, 2016, at 10:30 AM, Michal Tomaszewski <Michal.Tomaszewski@cca.pl> wrote:
Our several freeradius servers are using MySQL cluster (percona) and redis master/slave mapped to local port using HA Proxy on each freeradius server. So in fact each Freeradius sees 1 local port for MySQL and 1 local port for redis.
That's likely a source of the problem. If the HA proxy isn't closing bad connections, FreeRADIUS has no idea that the connection is bad,
When redis becomes unavailable for a while (restart of redis-server or simple restart of HA_Proxy) Freeradius does not check that connection is no longer available and keeps those invalid connections till lifetime time. In fact in such situation old connection should be discarded and new connection shall be opened.
How does FreeRADIUS know that the connection is no longer available? It doesn't periodically poll the idle connections. It just uses the connections when needed.
Standard lifetime in mods-available/redis is set to 86400 so redis module is not able to read/write anything from redis-server for 24 hours.
That doesn't make any sense. If FreeRADIUS tries to use a connection and the connection is bad, it closes the connection, and tries to open a new one. In addition, you should check the "idle_timeout" setting. If a connection has been idle for (say) 10 minutes, it may have timed out. Setting idle_timeout will ensure that the server doesn't even try old / idle connections, but instead closes them and opens a new one.
In /var/log/freeradius/radius.log there are only ERRORs when there is redis write command, there is no error information when there is (unsuccessful) read command. To see that debug mode shall be started. So in fact redis module is not verifying redis valid/invalid connection state.
FreeRADIUS just calls the redis library to access redis. If the library doesn't return errors on read, then there's little that FreeRADIUS can do.
Changing lifetime to e.g. 60 helps to force redis module start working (in 60 second from redis-server restart) but that is just a workaround not a solution.
Why not change idle_timeout?
#2 In var/log/freeradius/radius.log we have lots of errors: ERROR: SQL query failed: no connection
It seems that error shows up when connection is closed by e.g. idle-timeout... ... Thu Dec 22 15:59:45 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:47 2016 : Info: Need 0 more connections to reach min connections (5)
Those messages have been removed in the v3.0.x branch.
Thu Dec 22 15:59:47 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:48 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:48 2016 : ERROR: (164) ERROR: SQL query failed: no connection Thu Dec 22 15:59:48 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:51 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : ERROR: (172) ERROR: SQL query failed: no connection
I don't see that ERROR as much of a problem. The server tried to use an old connection, it failed, and when the server tried another connection, that one failed, too.
We tested to set freeradius use mysql directly, without haproxy and using default values in mods-availables/sql (including pool { start value) but it did not change this strange behavior.
Any help would be appreciated...
FreeRADIUS doesn't implement SQL, TCP, or any other networking code. It's at the mercy of the underlying libraries. If you take the SQL server down and the TCP connection remains up... blame the OS. Don't blame FreeRADIUS. It has no idea what's on the other end of the connection. Then set idle_timeout to a smaller value. That way idle connections will get closed more quickly, even if you have "lifetime" set to one day. Perhaps I could turn this problem around. What do you want FreeRADIUS to *do* when the SQL server goes down? How does FreeRADIUS know that SQL is down? What happens when SQL goes down? These aren't simple questions to answer. FreeRADIUS does it's best with the information it has. And in the end, if your SQL server goes down... of course FreeRADIUS will produce errors. There is simple no way for it to work around the fact that the SQL server is down. Alan DeKok.
Hi Alan, Thank you for a quick response.
On Dec 22, 2016, at 10:30 AM, Michal Tomaszewski <Michal.Tomaszewski@cca.pl> wrote:
Our several freeradius servers are using MySQL cluster (percona) and redis master/slave mapped to local port using HA Proxy on each freeradius server. So in fact each Freeradius sees 1 local port for MySQL and 1 local port for redis.
That's likely a source of the problem. If the HA proxy isn't closing bad connections, FreeRADIUS has no idea that the connection is bad,
Not exactly. It doesn't happen in case of MySQL. And please note - the same situation appears when you stop and start haproxy service. Connections are closed for sure as haproxy is stopped, so such connection does not exist any more. What more - there is no other way than haproxy to point freeradius to master redis-server while redis does not have another mechanism to check which server is a master and redirect requests. The haproxy use is the preferred one. It sends PING/PONG to check if redis server is alive. And in freeradius we have no other tools to choose master redis-server. Writes to slave are not replicated to master and thus are not persistent... https://discuss.pivotal.io/hc/en-us/articles/205309388-How-to-setup-HAProxy-... http://blog.haproxy.com/2014/01/02/haproxy-advanced-redis-health-check/ Freeradius does not check the state if the connection is alive. Maybe redis-tools (or redis library) is broken and does not give such possibility but you always can verify whether connection is alive using PING/PONG mechanism.
When redis becomes unavailable for a while (restart of redis-server or simple restart of HA_Proxy) Freeradius does not check that connection is no longer available and keeps those invalid connections till lifetime time. In fact in such situation old connection should be discarded and new connection shall be opened.
How does FreeRADIUS know that the connection is no longer available? It doesn't periodically poll the idle connections. It just uses the connections when needed.
Standard lifetime in mods-available/redis is set to 86400 so redis module is not able to read/write anything from redis-server for 24 hours.
That doesn't make any sense. If FreeRADIUS tries to use a connection and the connection is bad, it closes the connection, and tries to open a new one. Soi t is aparently not working this way...
In addition, you should check the "idle_timeout" setting. If a connection has been idle for (say) 10 minutes, it may have timed out. Setting idle_timeout will ensure that the server doesn't even try old / idle connections, but >instead closes them and opens a new one. So changing idl_timeout to 60 secs makes it working much better. But in for example MySQL connection state is checked much better.
In /var/log/freeradius/radius.log there are only ERRORs when there is redis write command, there is no error information when there is (unsuccessful) read command. To see that debug mode shall be started. So in fact redis module is not verifying redis valid/invalid connection state.
FreeRADIUS just calls the redis library to access redis. If the library doesn't return errors on read, then there's little that FreeRADIUS can do.
Maybe periodic PING/PONG on connection?
Changing lifetime to e.g. 60 helps to force redis module start working (in 60 second from redis-server restart) but that is just a workaround not a solution.
Why not change idle_timeout? I've changed it and it helps. But there is still 60-second delay to have redis-server responding.
#2 In var/log/freeradius/radius.log we have lots of errors: ERROR: SQL query failed: no connection
It seems that error shows up when connection is closed by e.g. idle-timeout... ... Thu Dec 22 15:59:45 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:47 2016 : Info: Need 0 more connections to reach min connections (5)
Those messages have been removed in the v3.0.x branch.
Yes. I saw this. Thank you. Tomorrow we'll compile the newest branch but it still does not resolve Error...
Thu Dec 22 15:59:47 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:48 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:48 2016 : ERROR: (164) ERROR: SQL query failed: no connection Thu Dec 22 15:59:48 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:51 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : Info: Need 0 more connections to reach min connections (5) Thu Dec 22 15:59:54 2016 : ERROR: (172) ERROR: SQL query failed: no connection
I don't see that ERROR as much of a problem. The server tried to use an old connection, it failed, and when the server tried another connection, that one failed, too. Yes. But connection has expired and probably something in connection's pool management can be improved.
We tested to set freeradius use mysql directly, without haproxy and using default values in mods-availables/sql (including pool { start value) but it did not change this strange behavior.
Any help would be appreciated...
FreeRADIUS doesn't implement SQL, TCP, or any other networking code. It's at the mercy of the underlying libraries. If you take the SQL server down and the TCP connection remains up... blame the OS. Don't blame FreeRADIUS. It has no idea what's on the other end of the connection.
Server is not down. Server is working perfectly all the time. Those messages are not because MySQL server is down. They are appearing in normal operation when server is 100% available. They are probably because idle_timeout closes the connection and freeradius does not record the fact that such connection does not exist.
Then set idle_timeout to a smaller value. That way idle connections will get closed more quickly, even if you have "lifetime" set to one day. Idle_timeout is in default value of 60 seconds. If I change it to e.g. 2 secs I have much more ERROR messages.
Perhaps I could turn this problem around. What do you want FreeRADIUS to *do* when the SQL server goes down? How does FreeRADIUS know that SQL is down? What happens when SQL goes down?
SQL is working all the time without any interruption. It is not this case.
These aren't simple questions to answer. FreeRADIUS does it's best with the information it has.
In my opinion connection is closed because of idle_timeout but freeradius still tries to use closed connection... ________________________________________ Uwaga: Treść niniejszej wiadomości może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. ________________________________ Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.If you have received this communication in error, please notify the sender immediately by replying to the message and deleting it from your computer. Thank you. ________________________________
On Dec 22, 2016, at 4:09 PM, Michal Tomaszewski <Michal.Tomaszewski@cca.pl> wrote:
That's likely a source of the problem. If the HA proxy isn't closing bad connections, FreeRADIUS has no idea that the connection is bad,
Not exactly. It doesn't happen in case of MySQL.
It doesn't matter to FreeRADIUS what the back-end database is.
And please note - the same situation appears when you stop and start haproxy service. Connections are closed for sure as haproxy is stopped, so such connection does not exist any more.
Then the OS is responsible for telling FreeRADIUS that the connection is closed. i.e. when FreeRADIUS tries to use the connection, it gets told "connection closed", and it tries another one. The server *may* keep an idle connection open, even if it's dead. That's because the server doesn't know it's dead.
What more - there is no other way than haproxy to point freeradius to master redis-server while redis does not have another mechanism to check which server is a master and redirect requests. The haproxy use is the preferred one. It sends PING/PONG to check if redis server is alive.
That's really the responsibility of the underling library. It's also bad practice to have FreeRADIUS to periodically "ping" every connection. There many be many connections, and if the server is idle, the pings serve no purpose.
And in freeradius we have no other tools to choose master redis-server. Writes to slave are not replicated to master and thus are not persistent...
Version 4 implements full Redis cluster support.
Freeradius does not check the state if the connection is alive.
If it's not using the connection, it doesn't matter if the connection is dead. if it is using the connection, it will discover when the connection is dead.
Maybe redis-tools (or redis library) is broken and does not give such possibility but you always can verify whether connection is alive using PING/PONG mechanism.
Then fix the library. We just don't have time to implement work-arounds for bugs in DB libraries.
So changing idl_timeout to 60 secs makes it working much better. But in for example MySQL connection state is checked much better.
Probably because the MySQL client library is smarter than the Redis one.
I've changed it and it helps. But there is still 60-second delay to have redis-server responding.
I have no idea why. Again, if the connection is dead, then the server discovers that immediately. If the connection *looks* alive but is really dead, then blame HA Proxy or the OS. They're responsible for closing connections.
I don't see that ERROR as much of a problem. The server tried to use an old connection, it failed, and when the server tried another connection, that one failed, too. Yes. But connection has expired and probably something in connection's pool management can be improved.
How? Make a concrete suggestion. Or send a patch.
Server is not down. Server is working perfectly all the time. Those messages are not because MySQL server is down. They are appearing in normal operation when server is 100% available. They are probably because idle_timeout closes the connection and freeradius does not record the fact that such connection does not exist.
No. And why guess? The source is available to you. The idle_timeout configuration is enforced *before* the SQL module gets a connection. The SQL module just asks for a connection, and then gets one.
Then set idle_timeout to a smaller value. That way idle connections will get closed more quickly, even if you have "lifetime" set to one day. Idle_timeout is in default value of 60 seconds. If I change it to e.g. 2 secs I have much more ERROR messages.
I don't see how that happens.
Perhaps I could turn this problem around. What do you want FreeRADIUS to *do* when the SQL server goes down? How does FreeRADIUS know that SQL is down? What happens when SQL goes down?
SQL is working all the time without any interruption. It is not this case.
These aren't simple questions to answer. FreeRADIUS does it's best with the information it has.
In my opinion connection is closed because of idle_timeout but freeradius still tries to use closed connection...
That's just not possible in the current code. Again, why guess? The code is available to you. Go read it. Alan DeKok.
Alan, Some more Information in the subject.
I don't see that ERROR as much of a problem. The server tried to use an old connection, it failed, and when the server tried another connection, that one failed, too. Yes. But connection has expired and probably something in connection's pool management can be improved.
How? Make a concrete suggestion. Or send a patch.
There is an ERROR in case the data is written to the MySQL correctly. As I understand this is not an ERROR situation. Error shall be thrown where there is no possibility to write data.
Server is not down. Server is working perfectly all the time. Those messages are not because MySQL server is down. They are appearing in normal operation when server is 100% available. They are probably because idle_timeout closes the connection and freeradius does not record the fact that such connection does not exist.
No. And why guess? The source is available to you.
I've checked the source and deeply checked the problem. The problem exists on Ubuntu 16.04 and MySQL client library 5.7 and newest build of freeradius. The same situation we can observe on our stage environment: freeradius 3.12 on Ubuntu 14.04 and MySQL client library 5.5.46. So suggested bug in the SQL library or its newer/older version seems to be not a real cause of the problem. There is 'ERROR: SQL query failed: no connection' ONLY in case of INSERTs or REPLACEs to Mysql database. Error is thrown but each INSERT and REPLACE is SUCCESSFUL. We made a lot of tests and: 1. EVERY (even simple) INSERT or REPLACE throws ERROR. MySQL is available during query. Insert or replace is successful. We made temporary table and tested inserts in accounting start/stop section: Tue Dec 27 14:42:02 2016 : Debug: (2) Executing select query: INSERT INTO debug_log (`data`) VALUES ('AAAAAAA') Tue Dec 27 14:42:02 2016 : ERROR: (2) SQL query failed: no connection 2. When you change insert query to select query - there are no errors. 3. Nor SELECT neither DELETE query throws error (to be sure we checked deletes with 0 rows deleted and 1 row deleted). Only INSERT and REPLACE. Each INSERT and each REPLACE. 3. There is also no error thrown when you change the insert query into: "INSERT INTO debug_log (`data`) VALUES ('AAAAAAA'); SELECT 1;" Examples: #1 real situation: Mon Nov 28 21:53:09 2016 : Debug: rlm_sql (sql): Reserved connection (10) Mon Nov 28 21:53:09 2016 : Debug: (12) Executing select query: INSERT INTO radius_radacct_closed ( acctsessionid, acctuniqueid, username, additional_id, device_id, realm, nasipaddress, nasportid, nasporttype, acctstarttime, acctupdatetime, acctstoptime, acctsessiontime, acctauthentic, connectinfo_start, connectinfo_stop, acctinputoctets, acctoutputoctets, calledstationid, callingstationid, acctterminatecause, servicetype, framedprotocol, framedipaddress ) VALUES ('583c98d000000001', '79ffcccafc3b54e87403444481bf65e5', 'z555', '555', '2', '', '192.168.176.1', '00000001', 'Wireless-802.11', FROM_UNIXTIME(1480366389 - 49), FROM_UNIXTIME(1480366389), FROM_UNIXTIME(1480366389), 49, '', '', '', '0' << 32 | '413029', '0' << 32 | '3533411', 'z5-d2', '90-B0-ED-E6-AE-15', 'User-Request', '', '', '192.168.176.2') Mon Nov 28 21:53:09 2016 : ERROR: (12) SQL query failed: no connection -- -- -- -- Mon Nov 28 21:54:45 2016 : Debug: rlm_sql (sql): Reserved connection (13) Mon Nov 28 21:54:45 2016 : Debug: (14) Executing select query: INSERT INTO radius_radacct_active ( acctsessionid, acctuniqueid, username, additional_id, device_id, realm, nasipaddress, nasportid, nasporttype, acctstarttime, acctstoptime, acctauthentic, connectinfo_start, connectinfo_stop, acctinputoctets, acctoutputoctets, calledstationid, callingstationid, servicetype, framedprotocol, framedipaddress ) VALUES ('583c996300000001', '3c96a44b069782db0d7076274139e521', 'z555', '5555', '2', '', '192.168.176.1', '00000001', 'Wireless-802.11', FROM_UNIXTIME(1480366485), FROM_UNIXTIME(1480366485 + 2700), '', '', '', '0' << 32 | '0', '0' << 32 | '0', 'z5-d2', '90-B0-ED-E6-AE-15', 'Login-User', '', '192.168.176.2') Mon Nov 28 21:54:45 2016 : ERROR: (14) SQL query failed: no connection #2 Test scenarios: Tue Dec 27 14:42:02 2016 : Debug: %{sql: INSERT INTO debug_log (`data`) VALUES ('AAAAAAA') } Tue Dec 27 14:42:02 2016 : Debug: Parsed xlat tree: Tue Dec 27 14:42:02 2016 : Debug: xlat --> sql Tue Dec 27 14:42:02 2016 : Debug: { Tue Dec 27 14:42:02 2016 : Debug: literal --> INSERT INTO debug_log (`data`) VALUES ('AAAAAAA') Tue Dec 27 14:42:02 2016 : Debug: } Tue Dec 27 14:42:02 2016 : Debug: %{User-Name} Tue Dec 27 14:42:02 2016 : Debug: Parsed xlat tree: Tue Dec 27 14:42:02 2016 : Debug: attribute --> User-Name Tue Dec 27 14:42:02 2016 : Debug: (2) EXPAND %{User-Name} Tue Dec 27 14:42:02 2016 : Debug: (2) --> z555 Tue Dec 27 14:42:02 2016 : Debug: (2) SQL-User-Name set to 'z555' Tue Dec 27 14:42:02 2016 : Debug: rlm_sql (sql): Reserved connection (1) Tue Dec 27 14:42:02 2016 : Debug: (2) Executing select query: INSERT INTO debug_log (`data`) VALUES ('AAAAAAA') **************************Tue Dec 27 14:42:02 2016 : ERROR: (2) SQL query failed: no connection Tue Dec 27 14:42:02 2016 : Debug: rlm_sql (sql): Released connection (1) Tue Dec 27 14:42:02 2016 : Info: Need 2 more connections to reach min connections (5) Tue Dec 27 14:42:02 2016 : Info: rlm_sql (sql): Opening additional connection (3), 1 of 29 pending slots used Tue Dec 27 14:42:02 2016 : Debug: rlm_sql_mysql: Starting connect to MySQL server Tue Dec 27 14:42:02 2016 : Debug: rlm_sql_mysql: Connected to database 'Radius' on 127.0.0.1 via TCP/IP, server version 5.6.34-79.1-56-log, protocol version 10 Tue Dec 27 14:42:02 2016 : Debug: (2) EXPAND %{sql: INSERT INTO debug_log (`data`) VALUES ('AAAAAAA') } Tue Dec 27 14:42:02 2016 : Debug: (2) --> Wed Dec 28 10:56:08 2016 : Debug: rlm_sql (sql): Reserved connection (0) Wed Dec 28 10:56:08 2016 : Debug: (0) Executing select query: INSERT INTO debug_log (data) VALUES ('CCCCCCCCCCCCCCCCCCCCCCC') **************************Wed Dec 28 10:56:08 2016 : ERROR: (0) SQL query failed: no connection -- Wed Dec 28 10:56:08 2016 : Debug: Parsed xlat tree: Wed Dec 28 10:56:08 2016 : Debug: attribute --> User-Name Wed Dec 28 10:56:08 2016 : Debug: (0) EXPAND %{User-Name} Wed Dec 28 10:56:08 2016 : Debug: (0) --> z555 Wed Dec 28 10:56:08 2016 : Debug: (0) SQL-User-Name set to 'z555' Wed Dec 28 10:56:08 2016 : Debug: rlm_sql (sql): Reserved connection (0) Wed Dec 28 10:56:08 2016 : Debug: (0) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA') **************************Wed Dec 28 10:56:08 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 11:11:15 2016 : Debug: literal --> INSERT INTO debug_log (data) VALUES ('BBBBBBBBBBBBBBB');SELECT 1 Wed Dec 28 11:11:15 2016 : Debug: (0) Executing select query: INSERT INTO debug_log (data) VALUES ('BBBBBBBBBBBBBBB');SELECT 1 Wed Dec 28 11:11:15 2016 : Debug: (0) EXPAND %{sql: INSERT INTO debug_log (data) VALUES ('BBBBBBBBBBBBBBB');SELECT 1 } Wed Dec 28 11:13:15 2016 : Debug: %{sql: INSERT INTO debug_log (data) VALUES ('CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 } Wed Dec 28 11:13:15 2016 : Debug: literal --> INSERT INTO debug_log (data) VALUES ('CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) EXPAND %{sql: INSERT INTO debug_log (data) VALUES ('CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 } Wed Dec 28 11:13:15 2016 : Debug: %{sql: INSERT INTO debug_log (data) VALUES ('FFFFFFFFFFF');SELECT 1} Wed Dec 28 11:13:15 2016 : Debug: literal --> INSERT INTO debug_log (data) VALUES ('FFFFFFFFFFF');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('FFFFFFFFFFF');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('FFFFFFFFFFF');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) EXPAND %{sql: INSERT INTO debug_log (data) VALUES ('FFFFFFFFFFF');SELECT 1} Wed Dec 28 11:13:15 2016 : Debug: %{sql: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 } Wed Dec 28 11:13:15 2016 : Debug: literal --> INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 In case of replace: Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Reserved connection (0) Wed Dec 28 15:14:01 2016 : Debug: (0) Executing select query: REPLACE INTO debug_log (id,data) VALUES (1,'CCCCCCCCCCCCCCCCCCCCCCC') **************************Wed Dec 28 15:14:01 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Released connection (0) Wed Dec 28 15:14:01 2016 : Info: Need 4 more connections to reach min connections (5) Wed Dec 28 15:14:01 2016 : Info: rlm_sql (sql): Opening additional connection (1), 1 of 31 pending slots used Wed Dec 28 15:14:01 2016 : Debug: rlm_sql_mysql: Starting connect to MySQL server Wed Dec 28 15:14:01 2016 : Debug: rlm_sql_mysql: Connected to database 'WiFi-Spot-Radius' on 127.0.0.1 via TCP/IP, server version 5.6.34-79.1-56-log, protocol version 10 Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{sql: REPLACE INTO debug_log (id,data) VALUES (1,'CCCCCCCCCCCCCCCCCCCCCCC') } Wed Dec 28 15:14:01 2016 : Debug: (0) --> Wed Dec 28 15:14:01 2016 : Debug: %{sql: REPLACE INTO debug_log (id,data) VALUES (3,'CCCCCCCCCCCCCCCCCCCCCCC') } Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: xlat --> sql Wed Dec 28 15:14:01 2016 : Debug: { Wed Dec 28 15:14:01 2016 : Debug: literal --> REPLACE INTO debug_log (id,data) VALUES (3,'CCCCCCCCCCCCCCCCCCCCCCC') Wed Dec 28 15:14:01 2016 : Debug: } Wed Dec 28 15:14:01 2016 : Debug: %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: attribute --> User-Name Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: (0) --> z555 Wed Dec 28 15:14:01 2016 : Debug: (0) SQL-User-Name set to 'z555' Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Reserved connection (0) Wed Dec 28 15:14:01 2016 : Debug: (0) Executing select query: REPLACE INTO debug_log (id,data) VALUES (3,'CCCCCCCCCCCCCCCCCCCCCCC') **************************Wed Dec 28 15:14:01 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Released connection (0) Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{sql: REPLACE INTO debug_log (id,data) VALUES (3,'CCCCCCCCCCCCCCCCCCCCCCC') } Wed Dec 28 15:14:01 2016 : Debug: (0) --> Wed Dec 28 15:14:01 2016 : Debug: %{sql: REPLACE INTO debug_log (id,data) VALUES (4,'CCCCCCCCCCCCCCCCCCCCCCC') } Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: xlat --> sql Wed Dec 28 15:14:01 2016 : Debug: { Wed Dec 28 15:14:01 2016 : Debug: literal --> REPLACE INTO debug_log (id,data) VALUES (4,'CCCCCCCCCCCCCCCCCCCCCCC') Wed Dec 28 15:14:01 2016 : Debug: } Wed Dec 28 15:14:01 2016 : Debug: %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: attribute --> User-Name Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: (0) --> z555 Wed Dec 28 15:14:01 2016 : Debug: (0) SQL-User-Name set to 'z555' Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Reserved connection (1) Wed Dec 28 15:14:01 2016 : Debug: (0) Executing select query: REPLACE INTO debug_log (id,data) VALUES (4,'CCCCCCCCCCCCCCCCCCCCCCC') **************************Wed Dec 28 15:14:01 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Released connection (1) Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{sql: REPLACE INTO debug_log (id,data) VALUES (4,'CCCCCCCCCCCCCCCCCCCCCCC') } Wed Dec 28 15:14:01 2016 : Debug: (0) --> Wed Dec 28 15:14:01 2016 : Debug: %{sql: REPLACE INTO debug_log (id,data) VALUES (5,'CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 } Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: xlat --> sql Wed Dec 28 15:14:01 2016 : Debug: { Wed Dec 28 15:14:01 2016 : Debug: literal --> REPLACE INTO debug_log (id,data) VALUES (5,'CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 Wed Dec 28 15:14:01 2016 : Debug: } Wed Dec 28 15:14:01 2016 : Debug: %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: attribute --> User-Name Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: (0) --> z555 Wed Dec 28 15:14:01 2016 : Debug: (0) SQL-User-Name set to 'z555' **************************Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Reserved connection (0) Wed Dec 28 15:14:01 2016 : Debug: (0) Executing select query: REPLACE INTO debug_log (id,data) VALUES (5,'CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Released connection (0) Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{sql: REPLACE INTO debug_log (id,data) VALUES (5,'CCCCCCCCCCCCCCCCCCCCCCC');SELECT 1 } Wed Dec 28 15:14:01 2016 : Debug: (0) --> 1 Wed Dec 28 15:14:01 2016 : Debug: %{sql: REPLACE INTO debug_log (id,data) VALUES (6,'CCCCCCCCCCCCCCCCCCCCCCC') } Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: xlat --> sql Wed Dec 28 15:14:01 2016 : Debug: { Wed Dec 28 15:14:01 2016 : Debug: literal --> REPLACE INTO debug_log (id,data) VALUES (6,'CCCCCCCCCCCCCCCCCCCCCCC') Wed Dec 28 15:14:01 2016 : Debug: } Wed Dec 28 15:14:01 2016 : Debug: %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: Parsed xlat tree: Wed Dec 28 15:14:01 2016 : Debug: attribute --> User-Name Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{User-Name} Wed Dec 28 15:14:01 2016 : Debug: (0) --> z555 Wed Dec 28 15:14:01 2016 : Debug: (0) SQL-User-Name set to 'z555' Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Reserved connection (1) Wed Dec 28 15:14:01 2016 : Debug: (0) Executing select query: REPLACE INTO debug_log (id,data) VALUES (6,'CCCCCCCCCCCCCCCCCCCCCCC') **************************Wed Dec 28 15:14:01 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 15:14:01 2016 : Debug: rlm_sql (sql): Released connection (1) Wed Dec 28 15:14:01 2016 : Debug: (0) EXPAND %{sql: REPLACE INTO debug_log (id,data) VALUES (6,'CCCCCCCCCCCCCCCCCCCCCCC') } #3 Same query log analysis: WITH ERROR: Wed Dec 28 11:01:33 2016 : Debug: %{sql: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA') } Wed Dec 28 11:01:33 2016 : Debug: Parsed xlat tree: Wed Dec 28 11:01:33 2016 : Debug: xlat --> sql Wed Dec 28 11:01:33 2016 : Debug: { Wed Dec 28 11:01:33 2016 : Debug: literal --> INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA') Wed Dec 28 11:01:33 2016 : Debug: } Wed Dec 28 11:01:33 2016 : Debug: %{User-Name} Wed Dec 28 11:01:33 2016 : Debug: Parsed xlat tree: Wed Dec 28 11:01:33 2016 : Debug: attribute --> User-Name Wed Dec 28 11:01:33 2016 : Debug: (0) EXPAND %{User-Name} Wed Dec 28 11:01:33 2016 : Debug: (0) --> z555 Wed Dec 28 11:01:33 2016 : Debug: (0) SQL-User-Name set to 'z555' Wed Dec 28 11:01:33 2016 : Debug: rlm_sql (sql): Reserved connection (1) Wed Dec 28 11:01:33 2016 : Debug: (0) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA') Wed Dec 28 11:01:33 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 11:01:33 2016 : Debug: rlm_sql (sql): Released connection (1) Wed Dec 28 11:01:33 2016 : Debug: (0) EXPAND %{sql: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA') } Wed Dec 28 11:01:33 2016 : Debug: (0) --> Wed Dec 28 11:01:33 2016 : Debug: (0) policy wifi-sessions.accounting { Wed Dec 28 11:01:33 2016 : Debug: (0) if (&Acct-Status-Type == start){ Wed Dec 28 11:01:33 2016 : Debug: (0) if (&Acct-Status-Type == start) -> FALSE Wed Dec 28 11:01:33 2016 : Debug: (0) elsif (&Acct-Status-Type == stop){ Wed Dec 28 11:01:33 2016 : Debug: (0) elsif (&Acct-Status-Type == stop) -> TRUE Wed Dec 28 11:01:33 2016 : Debug: (0) elsif (&Acct-Status-Type == stop) { WITHOUT ERROR (query modified to have select 1 on the end): Wed Dec 28 11:13:15 2016 : Debug: %{sql: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 } Wed Dec 28 11:13:15 2016 : Debug: Parsed xlat tree: Wed Dec 28 11:13:15 2016 : Debug: xlat --> sql Wed Dec 28 11:13:15 2016 : Debug: { Wed Dec 28 11:13:15 2016 : Debug: literal --> INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: } Wed Dec 28 11:13:15 2016 : Debug: %{User-Name} Wed Dec 28 11:13:15 2016 : Debug: Parsed xlat tree: Wed Dec 28 11:13:15 2016 : Debug: attribute --> User-Name Wed Dec 28 11:13:15 2016 : Debug: (1) EXPAND %{User-Name} Wed Dec 28 11:13:15 2016 : Debug: (1) --> z555 Wed Dec 28 11:13:15 2016 : Debug: (1) SQL-User-Name set to 'z555' Wed Dec 28 11:13:15 2016 : Debug: rlm_sql (sql): Reserved connection (1) Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: rlm_sql_mysql: Socket destructor called, closing socket Wed Dec 28 11:13:15 2016 : Debug: rlm_sql (sql): Reconnecting (1) Wed Dec 28 11:13:15 2016 : Debug: rlm_sql_mysql: Starting connect to MySQL server Wed Dec 28 11:13:15 2016 : Debug: rlm_sql_mysql: Connected to database 'Radius' on 127.0.0.1 via TCP/IP, server version 5.6.34-79.1-56-log, protocol version 10 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: rlm_sql (sql): Released connection (1) Wed Dec 28 11:13:15 2016 : Debug: (1) EXPAND %{sql: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 } Wed Dec 28 11:13:15 2016 : Debug: (1) --> 1 Wed Dec 28 11:13:15 2016 : Debug: (1) policy wifi-sessions.accounting { Wed Dec 28 11:13:15 2016 : Debug: (1) if (&Acct-Status-Type == start){ Wed Dec 28 11:13:15 2016 : Debug: (1) if (&Acct-Status-Type == start) -> FALSE Wed Dec 28 11:13:15 2016 : Debug: (1) elsif (&Acct-Status-Type == stop){ Wed Dec 28 11:13:15 2016 : Debug: (1) elsif (&Acct-Status-Type == stop) -> TRUE Wed Dec 28 11:13:15 2016 : Debug: (1) elsif (&Acct-Status-Type == stop) { As you can in second case see in place of: Wed Dec 28 11:01:33 2016 : ERROR: (0) SQL query failed: no connection Wed Dec 28 11:01:33 2016 : Debug: rlm_sql (sql): Released connection (1) after changing the query we have no ERROR message and in case of socket unavailable there is: Wed Dec 28 11:13:15 2016 : Debug: rlm_sql_mysql: Socket destructor called, closing socket Wed Dec 28 11:13:15 2016 : Debug: rlm_sql (sql): Reconnecting (1) Wed Dec 28 11:13:15 2016 : Debug: rlm_sql_mysql: Starting connect to MySQL server Wed Dec 28 11:13:15 2016 : Debug: rlm_sql_mysql: Connected to database 'Radius' on 127.0.0.1 via TCP/IP, server version 5.6.34-79.1-56-log, protocol version 10 Wed Dec 28 11:13:15 2016 : Debug: (1) Executing select query: INSERT INTO debug_log (data) VALUES ('AAAAAAAAAAAAAAAAAAAA');SELECT 1 Wed Dec 28 11:13:15 2016 : Debug: rlm_sql (sql): Released connection (1) what - I suppose - is normal and desired situation. Regards, Mike ________________________________________ Uwaga: Treść niniejszej wiadomości może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. ________________________________ Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.If you have received this communication in error, please notify the sender immediately by replying to the message and deleting it from your computer. Thank you. ________________________________
participants (3)
-
Alan DeKok -
Brian Candler -
Michal Tomaszewski