Recently, we rolled out a Shibboleth Single Sign On
service to protect one of our services. However, we started recieving
intermittant login failures, both on our automated monitoring and from
customers. Curiously these failures tended to happen mostly in the
evening, which isn’t a peak time for us. Debugging showed that the
authentication worked, but the authorisaton was failing. Shibboleth
works as an apache module and daemom that protects a service, which
communicates with a webservice that does the authenication processing.
The log files were showing an occasional SSL error in this communcation
link.
INFO shibtarget.SessionCache [43005] sessionGet: trying to get new attributes
for session (ID=_d0cd2f93840bb92050b28fa73d19ce4f)
INFO SAML.SAMLSOAPHTTPBinding [43005] sessionGet: sending SOAP message to
https://login.example.com/shibboleth/AA
ERROR SAML.SAMLSOAPHTTPBinding [43005] sessionGet: failed while contacting
SAML responder: error:1408F06B:SSL routines:SSL3_GET_RECORD:bad
decompression
ERROR shibtarget.SessionCache [43005] sessionGet: caught SAML exception
during SAML attribute query: SOAPHTTPBindingProvider::send() failed
while contacting SAML responder: error:1408F06B:SSL
routines:SSL3_GET_RECORD:bad decompression
ERROR shibtarget.SessionCache [43005] sessionGet: no response obtained
We didn’t manage to find any suitable solutions on the internet, so
we pulled out the trusty wireshark and started looking to see what was
going on. We could see that the client was advertising deflate and null
compression, and that the server was responding by asking for deflate
compression. However the client would then claim that there was a
decompression error in the servers response. This opened a few lines of
enquiry. I made sure that both ends of the connection were running the
same version of OpenSSL and they were both using 0.9.8a from Ubuntu
Dapper. Interestingly 0.9.8a is the first version that had compression
support. We found a couple of suggestions including forcing connections
to be SSL2, which lacked compression or recompiling openssl without zlib
support. As the former was easier, we tried that first by putting
SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
in /etc/apache2/mods-enabled/ssl.conf as suggested by Debian
bug #338008, and this seemed to work for around an hour. Packet
sniffing showed that it was still negotiating SSL3 including deflate
compression. Clearly we had to try something else.
Rather than recompile OpenSSL without zlib support, I thought I’d try
upgrading the version of OpenSSL to something later in case that fixed
the decompression bug. the version in Hardy is 0.9.8g, which sadly
required recompiling and disabling the Ubuntu change to enable
-Bsymbolic-functions during linking. Installing this on the
client end didn’t fix the problem, however installing it on the server
end seemed to fix it. So far it’s been running for 24 hours without an
error, so fingers crossed that this has fixed it for good.