"Shared secret is incorrect" - but it is identical!

[ken]

I'm trying to get FreeRadius working on a Fedora Core 6 server 
with a view to eventually using it to authenticate against 
Windows Active Directory via ntlm_auth for the Janet Roaming 
Service. The first attempts at configuring it failed rather 
drastically so I went back to the beginning and I'm doing things 
one step at at time, making one-line changes to configs then 
using radtest and/or radclient to  ensure it still works. I can 
now authenticate a users defined in users file, or in the Unix 
passwd file, from radtest on local machine. (i.e. the same one 
the server is running on). Next step is to check that I can use 
FreeRadius  over the network by trying radclient on another machine.

It doesn't work from the networked machine. I see the "invalid 
signature (err=2)!  (Shared secret is incorrect.)" message.

Debug log says to "double check the shared secret on the 
server". I have more than double checked it. I'm using the same 
shared secret on both machines.  I "know" the shared secret is 
correct because it works from the local machine.  But obviously 
it isn't! Because the encrypted password can't be read on the 
server. What can I do to make sure the shared secret truly is 
correct?

The definitions for both hosts are identical in the clients.conf 
file. At one point I  manually edited them to swap the names of 
servers while leaving the secrets the same, just in case there 
was some hidden unprintable character - but the new local one 
still worked, proving that the two entries in the clients.conf 
file are in fact identical.

The shared secrets used in the radtest command are identical. 
I'm cutting and pasting the *same* radtest command in, not 
retyping it.

To test for sure I put radclient commands in scripts on the 
remote machine, where they failed. Then I  ftped them from the 
machine they failed on to the  other one - where they worked! So 
it *has* to be the same!  And if I alter it in any way there 
then radtest fails so its not getting a free passage just 
because its local.

I have a horrid fear I've missed something totally obvious about 
how radclient works and that I'm doing something really really 
stupid stupid - but I can't see what. And I've been stuck here 
for over a week now. Any clues?

 From the local machine I get:

===================
[ken at monstera ~]$ /usr/local/bin/radtest -d /etc/raddb 
username at bbk.ac.uk password server.IP.addr 122 sharedsecret
Sending Access-Request of id 121 to server.IP.addr port 1812
         User-Name = "username at bbk.ac.uk"
         User-Password = "password"
         NAS-IP-Address = 255.255.255.255
         NAS-Port = 122
rad_recv: Access-Accept packet from host server.IP.addr:1812, 
id=121, length=20
===================

But when I try from the remote machine I get:

===================
  /usr/local/bin/radtest -d /etc/raddb username at bbk.ac.uk 
password server.IP.addr 122 sharedsecret
Sending Access-Request of id 184 to server.IP.addr port 1812
         User-Name = "username at bbk.ac.uk"
         User-Password = "password"
         NAS-IP-Address = 255.255.255.255
         NAS-Port = 122
rad_recv: Access-Reject packet from host server.IP.addr:1812, 
id=184, length=20
rad_verify: Received Access-Reject packet from client 
server.IP.addr port 1812 with invalid signature (err=2)! 
(Shared secret is incorrect.)
[ken at ficus ~]$ /usr/local/bin/radtest -d /etc/raddb 
username at bbk.ac.uk password server.IP.addr 122 sharedsecret
Sending Access-Request of id 246 to server.IP.addr port 1812
         User-Name = "username at bbk.ac.uk"
         User-Password = "password"
         NAS-IP-Address = 255.255.255.255
         NAS-Port = 122
rad_recv: Access-Reject packet from host server.IP.addr:1812, 
id=246, length=20
rad_verify: Received Access-Reject packet from client 
server.IP.addr port 1812 with invalid signature (err=2)! 
(Shared secret is incorrect.)
[ken at ficus ~]$ /usr/local/bin/radtest -d /etc/raddb 
username at bbk.ac.uk password server.IP.addr 122 sharedsecret
Sending Access-Request of id 7 to server.IP.addr port 1812
         User-Name = "username at bbk.ac.uk"
         User-Password = "password"
         NAS-IP-Address = 255.255.255.255
         NAS-Port = 122
rad_recv: Access-Reject packet from host server.IP.addr:1812, 
id=7, length=20
rad_verify: Received Access-Reject packet from client 
server.IP.addr port 1812 with invalid signature (err=2)! 
(Shared secret is incorrect.)
==================


I strongly suspect that I am doing something stupid on the 
client side, because the same request works from the local 
server. But just in case its relevant, on the server in debug 
mode the failed transaction looks like this:


==================
rad_recv: Access-Request packet from host client.IP.addr:32772, 
id=61, length=68
         User-Name = "username at bbk.ac.uk"
         User-Password = 
"V\303\245\321\364Fb\334\373\275\242\203\\o6\264"
         NAS-IP-Address = 255.255.255.255
         NAS-Port = 122
   Processing the authorize section of radiusd.conf
modcall: entering group authorize for request 9
   modcall[authorize]: module "preprocess" returns ok for request 9
radius_xlat: 
'/var/log/radius/radacct/client.IP.addr/auth-detail-20070703'
rlm_detail: 
/var/log/radius/radacct/%{Client-IP-Address}/auth-detail-%Y%m%d 
expands to 
/var/log/radius/radacct/client.IP.addr/auth-detail-20070703
   modcall[authorize]: module "auth_log" returns ok for request 9
   modcall[authorize]: module "chap" returns noop for request 9
   modcall[authorize]: module "mschap" returns noop for request 9
     rlm_realm: Looking up realm "bbk.ac.uk" for User-Name = 
"username at bbk.ac.uk"
     rlm_realm: Found realm "bbk.ac.uk"
     rlm_realm: Adding Stripped-User-Name = "username"
     rlm_realm: Proxying request from user username to realm 
bbk.ac.uk
     rlm_realm: Adding Realm = "bbk.ac.uk"
     rlm_realm: Authentication realm is LOCAL.
   modcall[authorize]: module "suffix" returns noop for request 9
   rlm_eap: No EAP-Message, not doing EAP
   modcall[authorize]: module "eap" returns noop for request 9
     users: Matched entry DEFAULT at line 20
   modcall[authorize]: module "files" returns ok for request 9
rlm_pap: WARNING! No "known good" password found for the user. 
Authentication may fail because of this.
   modcall[authorize]: module "pap" returns noop for request 9
modcall: leaving group authorize (returns ok) for request 9
   rad_check_password:  Found Auth-Type System
auth: type "System"
   Processing the authenticate section of radiusd.conf
modcall: entering group authenticate for request 9
   modcall[authenticate]: module "unix" returns notfound for 
request 9
modcall: leaving group authenticate (returns notfound) for request 9
auth: Failed to validate the user.
   WARNING: Unprintable characters in the password. ? 
Double-check the shared secret on the server and the NAS!
Delaying request 9 for 1 seconds
Finished request 9
Going to the next request
--- Walking the entire request list ---
Waking up in 1 seconds...
--- Walking the entire request list ---
Waking up in 1 seconds...
--- Walking the entire request list ---
Sending Access-Reject of id 61 to client.IP.addr port 32772
Waking up in 4 seconds...
--- Walking the entire request list ---
Cleaning up request 9 ID 61 with timestamp 468aaada
Nothing to do.  Sleeping until we see a request.
==================


And a successful one looks like this - the obvious difference is 
that the password is in clear  (though I have obfuscated it 
here) - as would be expected if there was no shared secret.

==================
rad_recv: Access-Request packet from host server.IP.addr:32770, 
id=170, length=46
         User-Name = "username"
         User-Password = "password"
   Processing the authorize section of radiusd.conf
modcall: entering group authorize for request 10
   modcall[authorize]: module "preprocess" returns ok for request 10
radius_xlat: 
'/var/log/radius/radacct/server.IP.addr/auth-detail-20070703'
rlm_detail: 
/var/log/radius/radacct/%{Client-IP-Address}/auth-detail-%Y%m%d 
expands to 
/var/log/radius/radacct/server.IP.addr/auth-detail-20070703
   modcall[authorize]: module "auth_log" returns ok for request 10
   modcall[authorize]: module "chap" returns noop for request 10
   modcall[authorize]: module "mschap" returns noop for request 10
     rlm_realm: No '@' in User-Name = "username", looking up 
realm NULL
     rlm_realm: Found realm "NULL"
     rlm_realm: Adding Stripped-User-Name = "username"
     rlm_realm: Proxying request from user username to realm NULL
     rlm_realm: Adding Realm = "NULL"
     rlm_realm: Authentication realm is LOCAL.
   modcall[authorize]: module "suffix" returns noop for request 10
   rlm_eap: No EAP-Message, not doing EAP
   modcall[authorize]: module "eap" returns noop for request 10
     users: Matched entry username at line 2
   modcall[authorize]: module "files" returns ok for request 10
rlm_pap: Found existing Auth-Type, not changing it.
   modcall[authorize]: module "pap" returns noop for request 10
modcall: leaving group authorize (returns ok) for request 10
   rad_check_password:  Found Auth-Type Local
auth: type Local
auth: user supplied User-Password matches local User-Password
   Processing the post-auth section of radiusd.conf
modcall: entering group post-auth for request 10
radius_xlat: 
'/var/log/radius/radacct/server.IP.addr/reply-detail-20070703'
rlm_detail: 
/var/log/radius/radacct/%{Client-IP-Address}/reply-detail-%Y%m%d 
expands to 
/var/log/radius/radacct/server.IP.addr/reply-detail-20070703
   modcall[post-auth]: module "reply_log" returns ok for request 10
modcall: leaving group post-auth (returns ok) for request 10
Sending Access-Accept of id 170 to server.IP.addr port 32770
Finished request 10
Going to the next request
--- Walking the entire request list ---
Waking up in 6 seconds...
--- Walking the entire request list ---
Cleaning up request 10 ID 170 with timestamp 468aab69
Nothing to do.  Sleeping until we see a
==================


Debug of startup looks like this (same in both cases obviously). 
I made new conf files to contain any local changes I might make 
& to yhelp me find my way aroudn radiusd.conf more easily - but 
they are just the sections of conf I might want to change pulled 
out and INCLUDEd back in so no substantial change:

==================
  /usr/local/sbin/radiusd -X -d /etc/raddb
Starting - reading configuration files ...
reread_config:  reading radiusd.conf
Config:   including file: /etc/raddb/bbk_fr_listen.conf
Config:   including file: /etc/raddb/bbk_fr_security.conf
Config:   including file: /etc/raddb/proxy.conf
Config:   including file: /etc/raddb/clients.conf
Config:   including file: /etc/raddb/snmp.conf
Config:   including file: /etc/raddb/eap.conf
Config:   including file: /etc/raddb/bbk_fr_mschap.conf
Config:   including file: /etc/raddb/bbk_fr_ldap.conf
Config:   including file: /etc/raddb/bbk_fr_passwd.conf
Config:   including file: /etc/raddb/bbk_fr_realms.conf
Config:   including file: /etc/raddb/bbk_fr_details.conf
Config:   including file: /etc/raddb/sql.conf
Config:   including file: /etc/raddb/bbk_fr_radutmp.conf
Config:   including file: /etc/raddb/bbk_fr_counters.conf
Config:   including file: /etc/raddb/bbk_fr_exec.conf
Config:   including file: /etc/raddb/bbk_fr_ippool.conf
  main: prefix = "/usr/local"
  main: localstatedir = "/var"
  main: logdir = "/var/log/radius"
  main: libdir = "/usr/local/lib"
  main: radacctdir = "/var/log/radius/radacct"
  main: hostname_lookups = no
  main: max_request_time = 30
  main: cleanup_delay = 5
  main: max_requests = 1024
  main: delete_blocked_requests = 0
  main: port = 0
  main: allow_core_dumps = no
  main: log_stripped_names = no
  main: log_file = "/var/log/radius/radius.log"
  main: log_auth = no
  main: log_auth_badpass = no
  main: log_auth_goodpass = no
  main: pidfile = "/var/run/radiusd/radiusd.pid"
  main: user = "(null)"
  main: group = "(null)"
  main: usercollide = no
  main: lower_user = "no"
  main: lower_pass = "no"
  main: nospace_user = "no"
  main: nospace_pass = "no"
  main: checkrad = "/usr/local/sbin/checkrad"
  main: proxy_requests = yes
  proxy: retry_delay = 5
  proxy: retry_count = 3
  proxy: synchronous = no
  proxy: default_fallback = yes
  proxy: dead_time = 120
  proxy: post_proxy_authorize = no
  proxy: wake_all_if_all_dead = no
  security: max_attributes = 200
  security: reject_delay = 1
  security: status_server = yes
  main: debug_level = 0
read_config_files:  reading dictionary
read_config_files:  reading naslist
Using deprecated naslist file.  Support for this will go away soon.
read_config_files:  reading clients
read_config_files:  reading realms
radiusd:  entering modules setup
Module: Library search path is /usr/local/lib
Module: Loaded exec
  exec: wait = yes
  exec: program = "(null)"
  exec: input_pairs = "request"
  exec: output_pairs = "(null)"
  exec: packet_type = "(null)"
rlm_exec: Wait=yes but no output defined. Did you mean output=none?
Module: Instantiated exec (exec)
Module: Loaded expr
Module: Instantiated expr (expr)
Module: Loaded PAP
  pap: encryption_scheme = "crypt"
  pap: auto_header = yes
Module: Instantiated pap (pap)
Module: Loaded CHAP
Module: Instantiated chap (chap)
Module: Loaded MS-CHAP
  mschap: use_mppe = yes
  mschap: require_encryption = yes
  mschap: require_strong = yes
  mschap: with_ntdomain_hack = yes
  mschap: passwd = "(null)"
  mschap: ntlm_auth = "/usr/bin/ntlm_auth --request-nt-key 
--username=%{Stripped-User-Name:-%{User-Name:-None}} 
--challenge=%{mschap:Challenge:-00} 
--nt-response=%{mschap:NT-Response:-00}"
Module: Instantiated mschap (mschap)
Module: Loaded System
  unix: cache = no
  unix: passwd = "(null)"
  unix: shadow = "(null)"
  unix: group = "(null)"
  unix: radwtmp = "/var/log/radius/radwtmp"
  unix: usegroup = no
  unix: cache_reload = 600
Module: Instantiated unix (unix)
Module: Loaded eap
  eap: default_eap_type = "peap"
  eap: timer_expire = 60
  eap: ignore_unknown_eap_types = no
  eap: cisco_accounting_username_bug = no
rlm_eap: Loaded and initialized type md5
rlm_eap: Loaded and initialized type leap
  gtc: challenge = "Password: "
  gtc: auth_type = "PAP"
rlm_eap: Loaded and initialized type gtc
  tls: rsa_key_exchange = no
  tls: dh_key_exchange = yes
  tls: rsa_key_length = 512
  tls: dh_key_length = 512
  tls: verify_depth = 0
  tls: CA_path = "(null)"
  tls: pem_file_type = yes
  tls: private_key_file = "/etc/raddb/certs/radius2.bbk.ac.uk.key"
  tls: certificate_file = "/etc/raddb/certs/radius2.bbk.ac.uk.pem"
  tls: CA_file = "/etc/raddb/certs/ct_root.pem"
  tls: private_key_password = "whatever"
  tls: dh_file = "/etc/raddb/certs/dh"
  tls: random_file = "/dev/urandom"
  tls: fragment_size = 1024
  tls: include_length = yes
  tls: check_crl = no
  tls: check_cert_cn = "(null)"
  tls: cipher_list = "(null)"
  tls: check_cert_issuer = "(null)"
rlm_eap_tls: Loading the certificate file as a chain
rlm_eap: Loaded and initialized type tls
  peap: default_eap_type = "mschapv2"
  peap: copy_request_to_tunnel = no
  peap: use_tunneled_reply = no
  peap: proxy_tunneled_request_as_eap = yes
rlm_eap: Loaded and initialized type peap
  mschapv2: with_ntdomain_hack = no
rlm_eap: Loaded and initialized type mschapv2
Module: Instantiated eap (eap)
Module: Loaded preprocess
  preprocess: huntgroups = "/etc/raddb/huntgroups"
  preprocess: hints = "/etc/raddb/hints"
  preprocess: with_ascend_hack = no
  preprocess: ascend_channels_per_line = 23
  preprocess: with_ntdomain_hack = no
  preprocess: with_specialix_jetstream_hack = no
  preprocess: with_cisco_vsa_hack = no
  preprocess: with_alvarion_vsa_hack = no
Module: Instantiated preprocess (preprocess)
Module: Loaded detail
  detail: detailfile = 
"/var/log/radius/radacct/%{Client-IP-Address}/auth-detail-%Y%m%d"
  detail: detailperm = 384
  detail: dirperm = 493
  detail: locking = no
Module: Instantiated detail (auth_log)
Module: Loaded realm
  realm: format = "suffix"
  realm: delimiter = "@"
  realm: ignore_default = no
  realm: ignore_null = no
Module: Instantiated realm (suffix)
Module: Loaded files
  files: usersfile = "/etc/raddb/users"
  files: acctusersfile = "/etc/raddb/acct_users"
  files: preproxy_usersfile = "/etc/raddb/preproxy_users"
  files: compat = "no"
Module: Instantiated files (files)
Module: Loaded Acct-Unique-Session-Id
  acct_unique: key = "User-Name, Acct-Session-Id, 
NAS-IP-Address, Client-IP-Address, NAS-Port"
Module: Instantiated acct_unique (acct_unique)
  detail: detailfile = 
"/var/log/radius/radacct/%{Client-IP-Address}/detail-%Y%m%d"
  detail: detailperm = 384
  detail: dirperm = 493
  detail: locking = no
Module: Instantiated detail (detail)
Module: Loaded radutmp
  radutmp: filename = "/var/log/radius/radutmp"
  radutmp: username = "%{User-Name}"
  radutmp: case_sensitive = yes
  radutmp: check_with_nas = yes
  radutmp: perm = 384
  radutmp: callerid = yes
Module: Instantiated radutmp (radutmp)
  detail: detailfile = 
"/var/log/radius/radacct/%{Client-IP-Address}/pre-proxy-detail-%Y%m%d"
  detail: detailperm = 384
  detail: dirperm = 493
  detail: locking = no
Module: Instantiated detail (pre_proxy_log)
  detail: detailfile = 
"/var/log/radius/radacct/%{Client-IP-Address}/post-proxy-detail-%Y%m%d"
  detail: detailperm = 384
  detail: dirperm = 493
  detail: locking = no
Module: Instantiated detail (post_proxy_log)
  detail: detailfile = 
"/var/log/radius/radacct/%{Client-IP-Address}/reply-detail-%Y%m%d"
  detail: detailperm = 384
  detail: dirperm = 493
  detail: locking = no
Module: Instantiated detail (reply_log)
Listening on authentication *:1812
Listening on accounting *:1813
Listening on proxy *:1814
Ready to process requests.

==================


  FreeRADIUS Version 1.1.5, for host i686-pc-linux-gnu, built on 
Mar  9 2007 at 15:07:40


The configurations are minimal:


relevant entries in clients file:

==================
client nnn.nnn.nnn.nnn {
     secret = sharedsecret
     shortname   = monstera
     nastype     = other
}
client nnn.nnn.nnn.nnn {
     secret = sharedsecret
     shortname   = ficus
     nastype     = other
}
==================

relevant entry in users file

==================
username Auth-Type := Local, User-Password == "password"
==================


As I said, authentication works for the host on which Freeradius 
is running, but not on the other.