Just recently we worked on encryption-at-rest with MongoDB for one of our Enterprise Advance customers. After we got the initial information from Thales CipherTrust Manager we ran into some problems. Please read here, how we fixed them.
Introduction
The error message "unable to get local issuer certificate" is nothing exclusive to MongoDB it rather stems from the SSL library you're using, most of the time this will be a locally installed version of openssl.
In general, this error message occurs if a certificate authority (an issuer) can't be verified. Our case was a bit different though but let's back up for a second and set up the playing field.
What is MongoDB Enterprise Advanced?
MongoDB comes in three flavours:
MongoDB Community Edition: a free version for everybody to use with a standard feature set
MongoDB Enterprise Advanced: a commercial version with an extended feature set for advanced use case like user authentication and authorization via LDAP, encryption-at-rest, and many more
MongoDB Atlas: a Database as a Service (DBaaS) offer
Encryption-at-rest with MongoDB Enterprise Advanced
Besides MongoDB Atlas encryption-at-rest in on-premise environments is possible only with MongoDB Enterprise Advanced. Here you have two options:
Use a static encryption key which resides in the local file system of the server to encrypt and decrypt the data
Use the Key Management Interoperability Protocol (KMIP) integration to get encryption keys from a proper Key Manager, e.g. Thales CipherTrust Manager
Please note: using a static key is only recommended for development or testing purposes, especially because the MongoDB Ops Manager won't run backups with a static key!
MongoDB Oops...no, Ops Manager? What is that?
Whenever you've acquired a licence to use MongoDB Enterprise Advance you are allowed to use the MongoDB Ops Manager. Which, very simply put, is your MongoDB Atlas on-premise. A Java application which automates, monitors and backups all your MongoDB processes.
Instead of maintaining configuration files yourself, the Ops Manager can do it for you.
Our problem
In our particular case we wanted to switch from encryption-at-rest via a static key file to using KMIP and Thales. According to this guide "Configuration on MongoDB", we were given a
client certificate
client key
domain ca
root ca
together with a server name and port.
After preparing the --kmipClientCertificateFile and --kmipServerCAFile accordingly, we modified the mongod configuration parameters via Ops Manager, added --kmipServerName and --kmipServerPort and applied the modified configuration.
Here we are, the mongod process wasn't able to start with the following error message: "SSL peer certificate validation failed: unable to get local issuer certificate".
The solution and helpful tools on the way
If you read the article until this point I will spare you more text and give you the answer right away:
The problem was different encryption algorithms in the certificate chain.
Which seems to make a difference in the KMIP integration of MongoDB but not so much for the openssl command we regularly use to check certificate chains manually before we implement necessary changes to MongoDB:
Test SSL handshake
openssl s_client -tls1_2 -connect [server name:port] -showcerts -verify 10 -CAfile [kmipServerCAFile] -cert [client cert] -key [client key]
This command was successful in the first attempt whereas MongoDB returned the above error.
Show server certificates
openssl s_client -tls1_2 -connect [server name:port] -showcerts
Analysing a given certificate
openssl x509 -in [certificate file] -noout -text
Conclusion
Given the above two commands we were able to spot the error, still it took us multiple runs at it. All the provided certificates and CAs used ECC while the KMIP server itself used RSA.
I hope this helps in any way. This kind of situation might be familiar to some database administrators and the solution might be obvious for the SSL nerds out there, it took us some time to figure it out though.
And because I couldn't find any other source pointing to this particular scenario I found it worthwhile recording it here.
Keep safe ;)
