Securing MongoDB with TLS | SSL on RHEL 9

Securing MongoDB with TLS on RHEL 9

Learn how to secure MongoDB on RHEL 9 using TLS / SSL—step-by-step instructions for certificate setup, configuring mongod, enforcing client authentication, and troubleshooting.

Table of Contents

🔈Introduction

Transport Layer Security (TLS) / SSL is essential to protect MongoDB traffic from eavesdropping and man-in-the-middle attacks. In this guide, you’ll learn how to enable TLS for mongod or mongos on RHEL 9, using both server certificates and optional client certificate validation.

✅ Why Use TLS / SSL for MongoDB?

  • Encrypts in-transit communications between clients and server
  • Verifies server identity via certificates
  • Enables optional X.509 client authentication
  • Helps satisfy compliance requirements (e.g. PCI, HIPAA)
  • Avoids “SSL handshake failed” and certificate trust errors
💡NOTE: MongoDB’s TLS support is built on the system OpenSSL library.

This tutorial assumes you already have a working MongoDB installation on RHEL 9, and basic knowledge of TLS / PKI.

▶️ Step 1: Prepare Certificates (Server & CA)

You need at least:

  • A Certificate Authority (CA) certificate and private key
  • A server certificate signed by that CA, including private key
  • Optionally, client certificates if you enforce client authentication

Here’s a minimal example using OpenSSL (for testing/dev). In production, use a proper CA or trusted provider.

				
					# Create CA private key and self-signed certificate
openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
    -subj "/C=US/ST=State/L=City/O=Org/OU=IT/CN=MyCA" \
    -out ca.pem

# Create server key and CSR
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key server.key -subj "/C=US/ST=State/L=City/O=Org/OU=DB/CN=db.example.com" -out server.csr

# Sign server CSR with CA
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
    -out server.crt -days 365 -sha256

# Combine server key + cert into a PEM file MongoDB can consume
cat server.key server.crt > mongodb.pem

# (Optional) Create client cert + key
openssl genpkey -algorithm RSA -out client.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key client.key -subj "/C=US/ST=State/L=City/O=Org/OU=Client/CN=client1" \
    -out client.csr
openssl x509 -req -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
    -out client.crt -days 365 -sha256
cat client.key client.crt > client.pem

🟢 Key points

  • The server certificate’s Common Name (CN) or Subject Alternative Name must match the hostname clients will connect to
  • Use strong key sizes (2048 / 4096)
  • For production, use OCSP, revocation lists, structured CA hierarchies

All PEM files should have secure file permissions (e.g. 600) and reside in a protected directory (e.g. /etc/ssl/mongo/).

▶️ Step 2: Configure mongod / mongos to Use TLS

On RHEL 9, the MongoDB package uses the mongod.conf configuration file. You can also pass flags manually, but config is more maintainable.

Here’s an example addition to /etc/mongod.conf:

				
					net:
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/ssl/mongo/mongodb.pem
    CAFile: /etc/ssl/mongo/ca.pem
    # disabledProtocols: TLS1_0,TLS1_1   # optional
    # CRLFile: /etc/ssl/mongo/crl.pem      # optional
security:
  # optional: enforce client certs (X.509)
  # but this must be consistent across replica set / cluster
  authorization: enabled
  # For internal cluster (replica sets / shards) auth:
  # clusterAuthMode: x509
SettingDescription
mode: requireTLSForces all connections to use TLS (no plaintext)
certificateKeyFilePath to server’s certificate + key
CAFileCertificate(s) to validate client certs
disabledProtocolsDisable weak protocols (e.g. TLS1.0 / 1.1)
CRLFileCertificate Revocation List, for blocked certs

If you prefer command-line flags, the equivalents are:

				
					mongod --tlsMode requireTLS \
  --tlsCertificateKeyFile /etc/ssl/mongo/mongodb.pem \
  --tlsCAFile /etc/ssl/mongo/ca.pem

Once configured, restart MongoDB:

				
					sudo systemctl restart mongod
				
			
				
					sudo journalctl -u mongod -f

Watch the logs for any TLS handshake or certificate errors.

▶️ Step 3: (Optional) Enforce Client Certificate Validation

If you require clients to present valid certificates, enable X.509 client authentication:

				
					net:
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/ssl/mongo/mongodb.pem
    CAFile: /etc/ssl/mongo/ca.pem
    allowConnectionsWithoutCertificates: false
security:
  clusterAuthMode: x509
  • allowConnectionsWithoutCertificates: false mandates clients present certs
  • You may include a CRLFile to block revoked client certs
  • Make sure client certificates are signed by the same CA used in CAFile

On the client side, connect like:

				
					mongosh --tls --host db.example.com:27017 \
  --tlsCAFile /etc/ssl/mongo/ca.pem \
  --tlsCertificateKeyFile /etc/ssl/mongo/client.pem \
  -u "CN=client1,OU=Client,O=Org,..." --authenticationMechanism MONGODB-X509

If you use multiple clients, issue distinct client certificates per user to allow selective revocation.

▶️ Step 4: Connect Clients Securely (CLI Example)

Assuming server is listening on db.example.com:27017:

				
					# Simple client (no client certificate)
mongosh --tls --host db.example.com:27017 \
  --tlsCAFile /etc/ssl/mongo/ca.pem

# With client cert + authentication
mongosh --tls --host db.example.com:27017 \
  --tlsCAFile /etc/ssl/mongo/ca.pem \
  --tlsCertificateKeyFile /path/to/client.pem \
  --authenticationMechanism MONGODB-X509 \
  -u "CN=client1,OU=Client,O=Org,C=US"

If you get a self signed certificate or Hostname/IP does not match certificate’s altnames error, you can—for testing—add:

  • --tlsAllowInvalidCertificates (bypass trust)
  • --tlsAllowInvalidHostnames (ignore hostname mismatch)

But never use these in production.

💡As many users on forums report, connection failures are often due to missing --tlsCertificateKeyFile or --tlsCAFile flags.

▶️ Step 5: Harden & Validate the Setup

🟢 Disable Weak Protocols & Ciphers

You can disallow TLS 1.0 and 1.1 negotiation:

				
					net:
  tls:
    disabledProtocols: TLS1_0,TLS1_1

As of MongoDB’s documentation, only strong ciphers (≥ 128 bits) are permitted by default.

🟢 Enable Online Certificate Rotation

MongoDB 5.0 and later supports online rotation of TLS / CA files without downtime. Confirm your version supports that if needed.

🟢 Monitor Logs

Check for handshake failures, certificate expiry warnings (Mongo logs warn if within 30 days).

🟢 Example TLS-Related Log Entries

				
					2025-10-11T10:22:33.123+0000 I  NETWORK  [connX] connection accepted from 203.0.113.5:56742 #Y (1 connection now open)
2025-10-11T10:22:33.124+0000 E  NETWORK  [connX] SSLHandshakeFailed: SSL peer certificate validation failed

📝 Common Pitfalls & Troubleshooting

SymptomLikely CauseSolution
“self signed certificate”Client does not trust CAUse correct --tlsCAFile, do not bypass trust in production
Hostname mismatchCN / SAN doesn’t match connection hostRegenerate server cert with correct SANs
“isMaster” network error or handshake failureMissing TLS flags or misconfigAdd --tlsCertificateKeyFile and --tlsCAFile to client
Connection success via IP but failure via DNSDNS name not in SANInclude both IP and DNS in SAN when generating cert
Expired certificate warningsCert near expirationRotate certs (or use online rotation)

Also, avoid redundancy between net.tls and net.ssl—MongoDB accepts either alias, but tls is preferred in recent versions.

📝 Full Sample mongod.conf (RHEL 9)

				
					# mongod.conf
storage:
  dbPath: /var/lib/mongo
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
processManagement:
  fork: false  # if using systemd
net:
  port: 27017
  bindIp: 0.0.0.0
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/ssl/mongo/mongodb.pem
    CAFile: /etc/ssl/mongo/ca.pem
    disabledProtocols: TLS1_0,TLS1_1
security:
  authorization: enabled
  # Optional (if using x509)
  # clusterAuthMode: x509

Ensure file permissions:

				
					chown mongod:mongod /etc/ssl/mongo/*.pem
				
			
				
					chmod 600 /etc/ssl/mongo/*.pem

Restart and monitor:

				
					sudo systemctl restart mongod
				
			
				
					sudo tail -F /var/log/mongodb/mongod.log

🏁 Conclusion

Securing MongoDB with TLS on RHEL 9 is a critical step toward ensuring the confidentiality and integrity of your database communications. Whether you’re operating a single instance or a complex replica set, enabling TLS/SSL protects data in transit from interception and tampering.

In this guide, we walked through the process of generating trusted certificates, configuring mongod to enforce encrypted connections, and optionally validating client identities using X.509 certificates. You also learned how to connect securely from the MongoDB shell and troubleshoot common TLS issues such as certificate mismatches and handshake failures.

By disabling outdated protocols, enforcing strict certificate validation, and regularly rotating credentials, you build a hardened MongoDB environment that aligns with modern security standards and compliance requirements.

With MongoDB properly secured over TLS, you can confidently scale your deployment knowing that your data is protected at every network boundary. Take time to automate certificate renewal and review your security posture periodically to stay ahead of evolving threats.

Did you find this article helpful? Your feedback is invaluable to us! Feel free to share this post with those who may benefit, and let us know your thoughts in the comments section below.

📕 Related Posts
Tags