Error: Received conflicting packet
Hi, all, Sometimes when there are too many requests from a NAS, like right after rebooting it and thus breaking current sessions, etc., freeradius 2.1.3 under FreeBSD begins loggin many many lines like this after the NAS re-sends unanswered packets: Error: Received conflicting packet from client 10.10.70.94 port 1646 - ID: 220 due to unfinished request 511166. Giving up on old request. I looked in src/main/event.c and found this code: default: gettimeofday(&when, NULL); when.tv_sec -= 1; /* * If the cached request was received * within the last second, then we * discard the NEW request instead of the * old one. This will happen ONLY when * the client is severely broken, and is * sending conflicting packets very * quickly. */ if (timercmp(&when, &request->received, <)) { radlog(L_ERR, "Discarding conflicting packet from " "client %s port %d - ID: %d due to recent request %d.", client->shortname, packet->src_port, packet->id, request->number); return 0; } received_conflicting_request(request, client); ^^^ request = NULL; break; Our authorization/accounting happens through rlm_perl and is written in Perl. Perhaps it's not fast enough to process many many requests in under 1 second (when.tv_sec), but aborting the current packet instead of the new duplicate one can hardly be justified. Please look at the line marked with ^^^ - it's where the error is logged and the current request is aborted, unless it was caught earlier by "Discarding conflicting packet", in which case the _new_ duplicate request is aborted, which is more correct. I propose that when.tv_sec be configurable in radiusd.conf, and not hardcoded like that.
rihad wrote:
Sometimes when there are too many requests from a NAS, like right after rebooting it and thus breaking current sessions, etc., freeradius 2.1.3 under FreeBSD begins loggin many many lines like this after the NAS re-sends unanswered packets:
Error: Received conflicting packet from client 10.10.70.94 port 1646 - ID: 220 due to unfinished request 511166. Giving up on old request.
The system is too slow to answer requests.
I looked in src/main/event.c and found this code: ... Our authorization/accounting happens through rlm_perl and is written in Perl. Perhaps it's not fast enough to process many many requests in under 1 second (when.tv_sec),
No, it *is* too slow to process requests. Buy a faster machine, do load balancing, or fix your Perl script to run faster.
but aborting the current packet instead of the new duplicate one can hardly be justified.
Nonsense. The duplicate one is an indication that the *NAS* has given up on the first packet. Spending more time processing the "current" packet is useless, because the NAS will ignore the Access-Accept for the old packet. "Fixing" FreeRADIUS to spend more time processing useless requests will only make the problem worse.
Please look at the line marked with ^^^ - it's where the error is logged and the current request is aborted, unless it was caught earlier by "Discarding conflicting packet", in which case the _new_ duplicate request is aborted, which is more correct.
No. You do not understand how RADIUS works. The code will NOT be changed to discard the new packet.
I propose that when.tv_sec be configurable in radiusd.conf, and not hardcoded like that.
No. There is no reason to "fix" the server. Alan DeKok.
Alan DeKok wrote:
but aborting the current packet instead of the new duplicate one can hardly be justified.
Nonsense. The duplicate one is an indication that the *NAS* has given up on the first packet. Spending more time processing the "current" packet is useless, because the NAS will ignore the Access-Accept for the old packet.
Absurd. The Dell PowerEdge 2950 w/ 2 quad-cores cannot itself without human intervention survive the "NAS attack" exactly due to having to give up on hundreds of requests per second not replied to in under 1 second, evidenced by an almost equal presence of many "Discarding conflicting packet" and "Received conflicting packet" lines in the log. That is, not many (if any) of our "Receved ..." lines are due to what could be considered a NAS timeout, and they should be treated like "Discarding ...", that is, the new request should be dropped.
"Fixing" FreeRADIUS to spend more time processing useless requests will only make the problem worse.
Please look at the line marked with ^^^ - it's where the error is logged and the current request is aborted, unless it was caught earlier by "Discarding conflicting packet", in which case the _new_ duplicate request is aborted, which is more correct.
No. You do not understand how RADIUS works. The code will NOT be changed to discard the new packet.
Perhaps someone more knowledgeable than you will be more able to assess all points involved.
I propose that when.tv_sec be configurable in radiusd.conf, and not hardcoded like that.
No. There is no reason to "fix" the server.
Alan DeKok.
rihad wrote:
Absurd. The Dell PowerEdge 2950 w/ 2 quad-cores cannot itself without human intervention survive the "NAS attack" exactly due to having to give up on hundreds of requests per second
Your dual quad-core box can't handle hundreds of packets a second? Wow... your Perl script is doing something *really* bad. Simple tests of the server with PAP to a MySQL database show a single core machine can get 1000's of requests/s. If there's no DB, FreeRADIUS can do 10's of 1000's of requests per second. Test it yourself without your Perl script. It's not hard to do.
not replied to in under 1 second, evidenced by an almost equal presence of many "Discarding conflicting packet" and "Received conflicting packet" lines in the log. That is, not many (if any) of our "Receved ..." lines are due to what could be considered a NAS timeout, and they should be treated like "Discarding ...", that is, the new request should be dropped.
OK... so you don't understand how RADIUS works. Conflicting packets *are* NAS timeouts. Nothing else causes them.
No. You do not understand how RADIUS works. The code will NOT be changed to discard the new packet.
Perhaps someone more knowledgeable than you will be more able to assess all points involved.
This is a joke, right? There are maybe 10 people on the planet who know as much about RADIUS than I do. I doubt very much that anyone knows *more* about RADIUS. Good luck solving your problem. It's clear you don't want expert help. Alan DeKok.
Alan DeKok wrote:
but aborting the current packet instead of the new duplicate one can hardly be justified.
Nonsense. The duplicate one is an indication that the *NAS* has given up on the first packet. Spending more time processing the "current" packet is useless, because the NAS will ignore the Access-Accept for the old packet.
Absurd. The Dell PowerEdge 2950 w/ 2 quad-cores cannot itself without human intervention survive the "NAS attack" exactly due to having to give up on hundreds of requests per second not replied to in under 1 second,
Nonsense. Dell Poweredge 1850 with single quad-core has survived 3000 simultaneous re-authentications on my network and dropped 5 (yes, just five in total) requests as mysql got overloaded. Your perl script is way too slow to deal with such situation. Don't blame freeradius, Dell or whatever for a problem that is exclusively of *your* making. If you have to use such a slow script you will have to use several radius server and load-balance requests between them.
That is, not many (if any) of our "Receved ..." lines are due to what could be considered a NAS timeout, and they should be treated like "Discarding ...", that is, the new request should be dropped.
No, NAS qouldn't wait on your script to finish so it gave up and has tried again with a *different* request. Reason for that is that it isn't waiting for a response for an initial request any more. Discarding the second packet and replying to inital one in such a situation would be - well - stupid and a waste of time. NAS will just ignore that reply - it gave up on that request already.
"Fixing" FreeRADIUS to spend more time processing useless requests will only make the problem worse.
Please look at the line marked with ^^^ - it's where the error is logged and the current request is aborted, unless it was caught earlier by "Discarding conflicting packet", in which case the _new_ duplicate request is aborted, which is more correct.
No. You do not understand how RADIUS works. The code will NOT be changed to discard the new packet.
Perhaps someone more knowledgeable than you will be more able to assess all points involved.
Oh, good luck with that one :-D I somehow doubt that you will find someone more knowledgeable than Alan on this matter. Ivan Kalik Kalik Informatika ISP
Ivan Kalik wrote:
That is, not many (if any) of our "Receved ..." lines are due to what could be considered a NAS timeout, and they should be treated like "Discarding ...", that is, the new request should be dropped.
No, NAS qouldn't wait on your script to finish so it gave up and has tried again with a *different* request. Reason for that is that it isn't waiting for a response for an initial request any more. Our radius-server timeout is high enough: 4 minutes. Once again: I suppose that what freeradius thinks of as "Received conflicting packet ..." are rather a bit delayed packets normally treated as "Discarding conflicting packet ...", i.e. they arrive at freeradius in maybe 1.01+ second after the first request, but freeradius drops the current request instead of the new one. Soon I'm gonna rebuild freeradius with changed tv_sec and check that.
OK, unless someone supports this idea the matter is closed for now. Many thanks to anyone.
Hi,
Our radius-server timeout is high enough: 4 minutes. Once again: I suppose that what freeradius thinks of as "Received conflicting packet ..." are rather a bit delayed packets normally treated as "Discarding conflicting packet ...", i.e. they arrive at freeradius in maybe 1.01+ second after the first request, but freeradius drops the current request instead of the new one. Soon I'm gonna rebuild freeradius with changed tv_sec and check that.
huh? do you not understand the basic context of this issue? if the NAS has sent a repeat RADIUS packet then it means that the original packet has already been timed out and the NAS should NOT accept an 'accept' response on that original packet. your RADIUS timeout is 4 minutes??? the highest sorts of values that you should ever expect a RADIUS request to be 'on the wire' is around 12 seconds - and thats for proxied packets that have travelled around the world. what OS are you running on that Dell? The only thing I can think of is one of the dodgy RedHat buggy releases where there was a MASSIVE Perl startup penalty - thats been fixed in all recent distros (unless its come back again). we have single and dual core Dell system (manufacturer really doesnt matter - we're still talking about basic PC hardware - that can do several thousand auths per second...and thats with Perl involved too. i wonder if you are killing your RADIUS via some others task - eg its the accounting thats blocking the threads because you are accounting to eg SQL rather than to disk and using buffered-sql (very very common issue) alan
Alan Buxey wrote:
Hi,
Our radius-server timeout is high enough: 4 minutes. Once again: I suppose that what freeradius thinks of as "Received conflicting packet ..." are rather a bit delayed packets normally treated as "Discarding conflicting packet ...", i.e. they arrive at freeradius in maybe 1.01+ second after the first request, but freeradius drops the current request instead of the new one. Soon I'm gonna rebuild freeradius with changed tv_sec and check that.
huh? do you not understand the basic context of this issue? if the NAS has sent a repeat RADIUS packet then it means that the original packet has already been timed out and the NAS should NOT accept an 'accept' response on that original packet. Please see the comment from the code snippet in src/main/event.c in my original posting. Some duplicate packets might arrive after 1 sec. by a slight margin, even though they logically are whatever that special-cased conditional was designed to handle.
Our radius-server timeout is high enough: 4 minutes. Once again: I suppose that what freeradius thinks of as "Received conflicting packet ..." are rather a bit delayed packets normally treated as "Discarding conflicting packet ...", i.e. they arrive at freeradius in maybe 1.01+ second after the first request, but freeradius drops the current request instead of the new one. Soon I'm gonna rebuild freeradius with changed tv_sec and check that.
huh? do you not understand the basic context of this issue? if the NAS has sent a repeat RADIUS packet then it means that the original packet has already been timed out and the NAS should NOT accept an 'accept' response on that original packet. Please see the comment from the code snippet in src/main/event.c in my original posting. Some duplicate packets might arrive after 1 sec. by a slight margin, even though they logically are whatever that special-cased conditional was designed to handle.
1 second? Freeradius keeps track of duplicates for 5 minutes by default. That is if processing has been completed. But in your case requests are still being processed 4 minutes after NAS sent them (if that is your retry interval on the NAS as you claim - default is usually 2 minutes). That is why it gave up on them and sent a new request. How do you think that adjusting that 1 second interval is going to help *your* case??? Stop hacking the server and start looking at your perl code. Do you really need to use it for authentication? Can you get all the data in authorize script and let freeradius default modules do the authentication (that can speed things up quite a bit)? Can you get (some of) the data using freeradius sql/ldap/whatever modules instead? Ivan Kalik Kalik Informatika ISP
Ivan Kalik wrote:
Our radius-server timeout is high enough: 4 minutes. Once again: I suppose that what freeradius thinks of as "Received conflicting packet ..." are rather a bit delayed packets normally treated as "Discarding conflicting packet ...", i.e. they arrive at freeradius in maybe 1.01+ second after the first request, but freeradius drops the current request instead of the new one. Soon I'm gonna rebuild freeradius with changed tv_sec and check that. huh? do you not understand the basic context of this issue? if the NAS has sent a repeat RADIUS packet then it means that the original packet has already been timed out and the NAS should NOT accept an 'accept' response on that original packet. Please see the comment from the code snippet in src/main/event.c in my original posting. Some duplicate packets might arrive after 1 sec. by a slight margin, even though they logically are whatever that special-cased conditional was designed to handle.
1 second? Freeradius keeps track of duplicates for 5 minutes by default. That is if processing has been completed. But in your case requests are still being processed 4 minutes after NAS sent them (if that is your retry interval on the NAS as you claim - default is usually 2 minutes). That is why it gave up on them and sent a new request. How do you think that adjusting that 1 second interval is going to help *your* case???
Trying for the third time: there are many, many requests of the "Discarding conflicting packet" kind, which for one reason or another are dupped by our Cisco NASes in under one second (see the code). And there are many, many lines of the "Received conflicting packet" fame (see the code). Now, it can be logically deduced that a big part of the latter are indeed of the former type (because none of the NASes have timeouts as low as 2-5 seconds). What I'd really love is for freeradius to stop killing the current request after receiving a dup 2-5 seconds apart. It's no problem for me to patch and rebuild freeradius myself, I just thought it wouldn't be fair not to share that idea with others.
Stop hacking the server and start looking at your perl code. Do you really need to use it for authentication? Can you get all the data in authorize script and let freeradius default modules do the authentication (that can speed things up quite a bit)? Can you get (some of) the data using freeradius sql/ldap/whatever modules instead?
The rlm_perl authorization/accounting is dealing with traffic shaping, so I'd rather fix this freeradius' shortcoming.
rihad wrote:
Trying for the third time:
Do you have any intention of reading the messages here?
there are many, many requests of the "Discarding conflicting packet" kind, which for one reason or another are dupped by our Cisco NASes in under one second (see the code). And there are many, many lines of the "Received conflicting packet" fame (see the code).
If (as you say) the NAS is sending a conflicting packet within 1 second of the first one, then the NAS is broken. It SHOULD wait 5 seconds for the old request to time out, before sending a new one that re-uses the same IP/port/Id. Take your NAS, and throw it in the garbage. Buy a real NAS that implements RADIUS.
Now, it can be logically deduced that a big part of the latter are indeed of the former type (because none of the NASes have timeouts as low as 2-5 seconds). What I'd really love is for freeradius to stop killing the current request after receiving a dup 2-5 seconds apart.
That won't solve anything. This has been explained to you. Did you understand the explanation, or are you simply ignoring it?
It's no problem for me to patch and rebuild freeradius myself, I just thought it wouldn't be fair not to share that idea with others.
Thanks. And we shared our opinions... and you told us we were wrong.
Stop hacking the server and start looking at your perl code. Do you really need to use it for authentication? Can you get all the data in authorize script and let freeradius default modules do the authentication (that can speed things up quite a bit)? Can you get (some of) the data using freeradius sql/ldap/whatever modules instead?
The rlm_perl authorization/accounting is dealing with traffic shaping, so I'd rather fix this freeradius' shortcoming.
That response completely ignores the question. Did you understand it? Alan DeKok.
Alan DeKok wrote:
rihad wrote:
Trying for the third time:
Do you have any intention of reading the messages here?
there are many, many requests of the "Discarding conflicting packet" kind, which for one reason or another are dupped by our Cisco NASes in under one second (see the code). And there are many, many lines of the "Received conflicting packet" fame (see the code).
If (as you say) the NAS is sending a conflicting packet within 1 second of the first one, then the NAS is broken. It SHOULD wait 5 seconds for the old request to time out, before sending a new one that re-uses the same IP/port/Id.
Take your NAS, and throw it in the garbage. Buy a real NAS that implements RADIUS.
Oh yeah? Isn't Cisco 7260 good enough for you?
Now, it can be logically deduced that a big part of the latter are indeed of the former type (because none of the NASes have timeouts as low as 2-5 seconds). What I'd really love is for freeradius to stop killing the current request after receiving a dup 2-5 seconds apart.
That won't solve anything. This has been explained to you.
Did you understand the explanation, or are you simply ignoring it?
Explained what? What good is an explanation without testing it? True, I still haven't tested increasing the 1 second wait either.
It's no problem for me to patch and rebuild freeradius myself, I just thought it wouldn't be fair not to share that idea with others.
Thanks. And we shared our opinions... and you told us we were wrong.
Stop hacking the server and start looking at your perl code. Do you really need to use it for authentication? Can you get all the data in authorize script and let freeradius default modules do the authentication (that can speed things up quite a bit)? Can you get (some of) the data using freeradius sql/ldap/whatever modules instead?
The rlm_perl authorization/accounting is dealing with traffic shaping, so I'd rather fix this freeradius' shortcoming.
That response completely ignores the question. Did you understand it?
Didn't I say the Perl code might not be fast enough to handle hundreds of requests per second? Probably up to several thousand. What good will some sql/ldap/whatever tests do if I'm not going to be using them? That said, I've increased cleanup_delay from 5 to 120, and max_requests from 10000 to 100000. Let's wait...
rihad wrote:
Oh yeah? Isn't Cisco 7260 good enough for you?
Q: Hi, I have a RADIUS server that is slower than a 386, and a NAS that violates the RADIUS protocol. What should I do? A: Fix the server and the NAS. Q: You bastards! How dare you tell me my equipment is broken! While this is entertaining, it only proves one thing. Alan DeKok.
Alan DeKok wrote:
rihad wrote:
Oh yeah? Isn't Cisco 7260 good enough for you?
Q: Hi, I have a RADIUS server that is slower than a 386, and a NAS that violates the RADIUS protocol. What should I do?
A: Fix the server and the NAS.
Q: You bastards! How dare you tell me my equipment is broken!
While this is entertaining, it only proves one thing.
Being 100% correct protocol-wise means nothing, if the software can't fit well into an environment. Just a recent example off the top of my head: dnscache. Its security and DNS protocol support are astonishing. But due to it being unable to work reasonably under certain circumstances, and due to its author preferring to stick with the arguable RFC specification which is outright buggy, I was forced to go back to BIND. In reply to my post Jeremy Kister was kind enough to describe the problem very well: http://marc.info/?l=djbdns&m=125265930702615&w=2 and linked from above: http://securepoint.com/lists/html/djbdns/2007-01/msg00033.html
rihad wrote:
Being 100% correct protocol-wise means nothing, if the software can't fit well into an environment.
So you agree that the NAS is broken.
Just a recent example off the top of my head: dnscache. Its security and DNS protocol support are astonishing. But due to it being unable to work reasonably under certain circumstances, and due to its author preferring to stick with the arguable RFC specification which is outright buggy, I was forced to go back to BIND. In reply to my post Jeremy Kister was kind enough to describe the problem very well: http://marc.info/?l=djbdns&m=125265930702615&w=2 and linked from above: http://securepoint.com/lists/html/djbdns/2007-01/msg00033.html
Why do you insist that FreeRADIUS is at fault, when you have already been told that the NAS is broken? Why are you not calling Cisco, and asking them to fix their broken NAS? Do you hate FreeRADIUS? Are you interested in blaming it for every problem in the network? And your argument about RFC compliance is made to the wrong person. There's a RADIUS RFC called "issues and fixes", which points out problems with older RFC's, and says how they should be fixed. Guess who's co-author? There's another RFC that's up and coming, called "guidelines", about how to write RADIUS RFC's so that they can easily be implemented. Guess who's co-author? And have you looked at the FreeRADIUS source code? There are things in the RFC's that it *doesn't* do, because they're stupid. There are other things that are *now* in RFC's, because FreeRADIUS did them first. Sorry... your debug messages show two things: slow Perls scripts, and a buggy NAS. Nothing else. You have *not* identified bad behavior in FreeRADIUS, unlike others recently on this list. Alan DeKok.
Being 100% correct protocol-wise means nothing, if the software can't fit well into an environment.
Exactly. The only problem being your inability to comprehend that freeradius is not faulty but it is your perl script that can't cope. If you are unwilling to alter it in order to speed things up you can separate authentication and accounting using buffered-sql virtual server. On high loads that arrangement will favour authentication while accounting will lag behind and catch up when load goes down. You shouldn't have any conflicting packets then because router will just be re-sending accounting packets. Ivan Kalik Kalik Informatika ISP
Ivan Kalik wrote:
Being 100% correct protocol-wise means nothing, if the software can't fit well into an environment.
Exactly. The only problem being your inability to comprehend that freeradius is not faulty but it is your perl script that can't cope. Why do you not understand that even if I put "sleep 1" right before finishing a request in my auth/acct Perl scripts, meaning each request would take at least 1 second to process, freeradius shouldn't care! It shouldn't be canceling the current request if another packet arrives a couple of seconds later! Being swamped by requests it SHOULD be able to make progress, and not be stomping on its toes canceling current requests without any progress. And for that when.tv_sec = 1 should be a bit higher, so it SHOULD be configurable, because some poor soul might still prefer 2-5 second NAS timeout.
rihad wrote:
Ivan Kalik wrote:
Exactly. The only problem being your inability to comprehend that freeradius is not faulty but it is your perl script that can't cope. Why do you not understand that even if I put "sleep 1" right before finishing a request in my auth/acct Perl scripts, meaning each request would take at least 1 second to process, freeradius shouldn't care!
Everyone understands that. No one *cares*. It's not *relevant*.
It shouldn't be canceling the current request if another packet arrives a couple of seconds later!
Yes, it should. You've been told why. If you don't understand the explanation, go read the messages again.
Being swamped by requests it SHOULD be able to make progress,
... working on requests that it will send to the NAS, and which the NAS will ignore. It's like asking a web server to continue processing a client request after the client has closed the TCP connection. After all, the web server is still making progress! Let's waste more CPU time rendering content that will *NEVER* make it to the client! That's a dumb idea.
and not be stomping on its toes canceling current requests without any progress. And for that when.tv_sec = 1 should be a bit higher, so it SHOULD be configurable, because some poor soul might still prefer 2-5 second NAS timeout.
If your NAS is transmitting a two packets with the same source IP, port, RADIUS code, and ID within 1 second... it's broken. No amount of "fixing" FreeRADIUS will change that. No amount of outright denial will change that. No amount of ignoring our explanations will change that. Now stop arguing on this list. It is impossible to convince us that wasting CPU time is a good idea. (And ignoring our explanations makes it clear that you have no interest in educating yourself) You have the source code to the server: butcher it to do whatever you want. Alan DeKok.
Hi,
finishing a request in my auth/acct Perl scripts, meaning each request would take at least 1 second to process, freeradius shouldn't care! It
okay...you have a daemon listening on port 1812 ... how many threads or radiusd processes are you running - because , for example, if you have 20 then once you get 100 requests, only the first 20 can be answered.. because all threads are stuck with your 1 second delay... thats 80 requests lost on the wire. those will be sent again.....then you need to deal with the ones already being dealt with. if you put anything into the way that will slow down the ability to answer queries, then it will mess things up for you. if your PERL is the problem, then increase number of threads/children so you can handle more.... we have 64 listeners on our system that uses live PERL for dealing with requests...otherwise there were not enough listeners. ...if its accounting thats getting in the way (slow DB insert/updates) then move accounting out of the way - use the buffered-sql virtual server to deal with them 'out of band'. my systems can deal with hundreds of access requests per second...and are literally flooded by that many accounting-requests per second all day long. we are a cisco shop (and plenty of people on this list have had to deal with cisco wierdnesses in the world of RADIUS - there are workarounds all over the place for things they do wrong) and our solution didnt 'just work' overnight - because of loading issues, because of tweaks needed - we have FreeRADIUS because we could do these things to get around NAS limitations, NAS bugs and people NOT respecting the RFCs I'd rather have a server that DID follow the RFCs than one made by the vendor who chose to break the RFCs in their products int he first place. alan
Rihad,
Take your NAS, and throw it in the garbage. Buy a real NAS that implements RADIUS.
Oh yeah? Isn't Cisco 7260 good enough for you?
Hmmm ... A few months ago I was working on a project with a Cisco 72XX terminating PPoE connections from DSL modems. I was using custom SQL code during the authentication process. When the PPoE sessions were reset on the Cisco 72XX to test a power failure, there was a problem authenticating the PPoE sessions. Once I cleaned up the SQL code to improve its performance, the authentication problems went away. FreeRADIUS, MySQL with custom SQL code and the Cisco 72XX were able to authenticate over 14,000 PPoE sessions in 30 seconds. We also had a problem with broken DSL modems that continuously resent authentication requests with bad credentials. Based on my experience with the above project and many other projects, FR can process 1,000s of authenticate requests per second and performance problems tend be fixed by improving the performance of custom code, optimizing a database schema and/or tuning the database. You are welcome to tweak the FR server but based on the experience of many people running really big networks, that will not solve the problem. If you provide more information about your network and what you are trying to accomplish, people on the list can help you. Provide the following: - Perl script - Description of your network and what you are trying to accomplish with the PERL script - Copy of your configuration file The more information the better. Tim
participants (5)
-
Alan Buxey -
Alan DeKok -
Ivan Kalik -
rihad -
Tim Sylvester