rants, tirades, ruminations

As a semi-conscientious netizen, I feel it’s my duty to post about the insanity of dealing with Java keystore files when you already have X.509 PEM encoded certificates and intermediate CA certificates. I spent multiple hours over the last few days trying to grok this mess and I never want to spend another moment of my life trying to reinvent the wheel when I have to do this again several years from now.

Like most server administrators probably, I have an existing set of signed server certificates along with a bundle of CA signed intermediate certificates, all in X.509 PEM format (those base64 encoded ASCII text files that everyone knows and loves in the Unix world; if you’re using Windows and have PKCS #12 encoded files, you’ll need to look up how to convert them using the openssl command). But now I need to deploy something Java based (often Tomcat applications) which requires a Java keystore file instead of the much saner X.509 PEM format that practically everything that isn’t Java uses without any problems. This is where the insanity starts. And yes, I realize that newer versions of Tomcat can use OpenSSL directly which allows you to use X.509 PEM encoded files directly also, but that wasn’t an option here. And yes, I also realize that you could do some crazy wrapper setup using Apache on the front and the Tomcat application on the back. But that’s ludicrous just to work around how idiotic Java applications are about handling SSL certificates.

Every other piece of Unix software I’ve ever configured expects a server certificate and private key and possibly a single or even multiple intermediate certificates to enable SSL or TLS functionality. Granted, some applications are better about explicitly supporting intermediate certificates. But even ones that don’t almost always allow you to concatenate all of your certificates together (in order from least to most trusted; so, server certificate signed by some intermediate signed by possibly another intermediate signed by a self-signed CA certificate, where the top level CA certificate is normally left off of the chain). The point is, the end client ends up getting said blob and can then check that the last intermediate certificate is signed by a locally trusted top level CA certificate already present on the client’s device.

All of the documentation I could find says to import the intermediate and CA certificates into the keystore using the -trustcacerts option and using different aliases. The problem I was seeing though was that testing the validity of my server’s certificate after I installed the keystore this way using OpenSSL’s s_client always resulted in the server certificate not validating. Looking at s_client with -showcerts enabled, all I was ever getting back from the server during the initial SSL handshake was the lone server certificate without any of the intermediate certificates, unlike any of my other Apache or nginx server where the entire certificate blob was being passed from the server to the client, allowing s_client to verify that the certificate was in fact trusted by my local CA bundle installed as part of my operating system. If you want to try validating your own server’s certificate, use something like:

openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -showcerts -connect www.bitgnome.net:443

I finally ran across a post which mentioned keystore as part of the vtcrypt project. This turned out to be the key in making everything work the way I normally expect them to work.

Now before you run off to do the magic below, you will need to convert your PKCS #8 PEM formatted private key into a DER formatted key. You will need to do something like:

openssl pkcs8 -topk8 -nocrypt -outform DER -in server.key -out server.pkcs8

The handy thing about keystore is that it will ingest a standard X.509 PEM encoded certificate file, even when it has multiple certificates present, and spit out that desperately needed Java keystore with an alias that actually has multiple ceritficates present also! I include the magic here for demonstration purposes:

~/vt-crypt-2.1.4/bin/keystore -import -keystore test.jks -storepass changeit -alias tomcat -cert server+intermediate.crt -key server.pkcs8

That’s it! The test.jks keystore doesn’t need to exist. This will create it. Check to make sure that the keystore now contains the correct information:

~/vt-crypt-2.1.4/bin/keystore -list -keystore test.jks -storepass changeit

and you should see your certificate chain starting with your server certificate and ending with your last intermediate certificate. Once I installed the keystore in my application, s_client was able to successfully verify the now complete chain of trust from server certificate to my locally trusted CA root certificate.