TLS/SSL Configuration for Servers

TLS (Transport Layer Security) and its deprecated predecessor SSL (Secure Sockets Layer) define the cryptographic protocols that protect data transmitted between servers and clients. Correct configuration of these protocols is a mandatory technical control under federal security frameworks and multiple sector-specific compliance regimes. Misconfiguration — including the use of deprecated protocol versions, weak cipher suites, or improperly managed certificates — represents one of the most consistently exploited attack surfaces in server infrastructure.

Definition and scope

TLS is a cryptographic protocol that establishes authenticated, encrypted communication channels between networked endpoints. The Internet Engineering Task Force (IETF) governs the specification through published RFCs; TLS 1.3 is defined in RFC 8446, published in August 2018. TLS 1.2, defined in RFC 5246, remains in supported use with properly restricted cipher configurations, while TLS 1.0 and 1.1 are formally deprecated by RFC 8996.

SSL, in all versions (2.0 and 3.0), is cryptographically broken and has no legitimate deployment context in contemporary infrastructure. The scope of TLS/SSL configuration covers web servers, mail servers, database connection layers, API gateways, load balancers, internal service-to-service communication, and VPN endpoints. Each layer has distinct configuration requirements and certificate management responsibilities.

Regulatory scope is substantial. The PCI DSS standard (currently version 4.0, published by the PCI Security Standards Council) explicitly prohibits TLS 1.0 and requires TLS 1.1 to be disabled except under defined migration conditions (PCI DSS v4.0, Requirement 4.2.1). NIST SP 800-52 Revision 2, Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations, governs federal agency deployments and defines approved cipher suites and certificate requirements (NIST SP 800-52r2). HIPAA's technical safeguard requirements under 45 CFR § 164.312(e) mandate encryption of data in transit, which is operationalized through TLS deployment.

For a broader view of how encryption standards integrate with server infrastructure protection, see Server Encryption at Rest and in Transit and Server Certificate and PKI Management.

How it works

TLS operates through a handshake process that precedes any application-layer data exchange. The steps below describe the TLS 1.3 handshake, which reduces round trips compared to TLS 1.2:

  1. ClientHello — The client transmits supported protocol versions, cipher suites, and a random value. TLS 1.3 also includes key share data in this message.
  2. ServerHello — The server selects a cipher suite, returns its key share, and begins the certificate chain transmission.
  3. Certificate and CertificateVerify — The server presents its X.509 certificate, which the client validates against a trusted Certificate Authority (CA) root store. The server proves possession of the private key via a digital signature.
  4. Finished — Both parties derive session keys from the exchanged key material using the negotiated key exchange algorithm (ECDHE in TLS 1.3). The handshake completes and encrypted data transfer begins.
  5. Application Data — All subsequent traffic is encrypted with symmetric session keys and authenticated with AEAD (Authenticated Encryption with Associated Data) algorithms such as AES-256-GCM or ChaCha20-Poly1305.

TLS 1.3 eliminates static RSA key exchange, making forward secrecy mandatory for all sessions. TLS 1.2 permits non-forward-secret cipher suites (e.g., RSA key exchange), which NIST SP 800-52r2 explicitly discourages for new deployments. The distinction is operationally significant: without forward secrecy, a future compromise of a server's private key can decrypt previously captured session traffic.

Cipher suite configuration is a discrete administrative act. The Center for Internet Security (CIS) publishes server benchmarks — including for Apache HTTP Server, NGINX, and Microsoft IIS — that enumerate recommended and prohibited cipher strings (CIS Benchmarks).

Common scenarios

HTTPS termination at a web server or load balancer is the most common TLS deployment. In this configuration, the server holds the private key and certificate; the client browser validates the certificate chain. Misconfiguration here — including expired certificates, incomplete certificate chains, or weak cipher support — results in browser trust failures and potential downtime. The profile for web server TLS configuration connects directly to Web Server Security Configuration.

Database connection encryption applies TLS to connections between application servers and database engines. MySQL, PostgreSQL, and Microsoft SQL Server each have native TLS support. Organizations operating under PCI DSS scope must enforce encrypted connections at this layer; unencrypted internal database connections remain a common audit finding.

Mail server STARTTLS and SMTPS present a distinct scenario. STARTTLS upgrades a plaintext SMTP connection opportunistically; SMTPS (port 465) requires TLS from connection initiation. The Internet Society and RFC 8314 recommend SMTPS over STARTTLS for submission services. STARTTLS-based configurations are vulnerable to downgrade attacks if DANE (DNS-Based Authentication of Named Entities) or MTA-STS is not deployed. See Mail Server Security for sector-specific requirements.

Mutual TLS (mTLS) extends standard TLS by requiring both client and server to present X.509 certificates. This pattern is standard in zero trust architectures and service mesh deployments. The mTLS configuration burden is higher due to client certificate provisioning, but it eliminates reliance on credential-only authentication between services. The relationship between mTLS and broader access models is covered under Zero Trust Architecture for Servers.

Decision boundaries

TLS version selection follows a clear hierarchy based on regulatory and standards mandates:

Certificate validity periods became a distinct decision boundary after the CA/Browser Forum reduced the maximum accepted validity for publicly trusted TLS certificates to 398 days, a change that took effect in September 2020. Automation via ACME protocol (as used by Let's Encrypt, operated by the Internet Security Research Group) is the standard operational response to this constraint.

The choice between single-domain, multi-domain (SAN), and wildcard certificates involves coverage scope and key compromise blast radius. A compromised wildcard private key exposes all subdomains under that certificate, a risk that enterprise PKI governance frameworks typically mitigate by restricting wildcard use to isolated environments. Server Certificate and PKI Management details certificate lifecycle governance.

Organizations subject to FIPS 140-2 or FIPS 140-3 validation requirements — including federal agencies and contractors — must ensure TLS implementations use only FIPS-validated cryptographic modules, as specified by NIST through the Cryptographic Module Validation Program (CMVP). This constrains cipher suite selection to FIPS-approved algorithms and may require specific TLS library configurations beyond default OS settings.

References

Explore This Site

Regulations & Safety Regulatory References Tools & Calculators Password Strength Calculator