TLS Handshake

The TLS (Transport Layer Security) handshake is a multi-step protocol used to establish a secure communication channel between a client and a server over an insecure network (like the Internet). It builds on top of TCP and ensures:

• Confidentiality (encryption)
• Integrity (data not altered)
• Authentication (server identity verified)

TLS is most commonly used in HTTPS, secure email, VPNs, and secure APIs.

TLS Handshake – Step-by-Step Breakdown

Default Modern TLS 1.2+ (simplified)

Who	Message	Purpose
Client	ClientHello	Proposes supported TLS versions, cipher suites, and sends random value.
Server	ServerHello	Selects TLS version, cipher suite, and sends its certificate and random value.
Server	Certificate	Contains server’s public key signed by a trusted CA.
Client	Verify certificate	Checks if the cert is valid (via CA chain and hostname).
Client	Pre-Master Key	Encrypts and sends a pre-master secret using server’s public key.
Both	Generate session keys	Both generate the same symmetric key using the shared secret + randoms.
Client	Finished	Sends encrypted handshake message.
Server	Finished	Sends encrypted handshake message.
Both	Secure communication begins	Now both sides use symmetric encryption.

Purpose of TLS in System Design

Goal	How TLS Handshake Helps
Authentication	Client verifies it’s talking to a real, trusted server (via certificate).
Encryption	All HTTP data is encrypted using symmetric keys generated during handshake.
Integrity	Protects against tampering or replay attacks using MACs or AEAD encryption.
Forward secrecy	(with ephemeral key exchange like ECDHE) Even if private key is stolen, past sessions are safe.

Visual Diagram of TLS Handshake

Client                         Server
  |                             |
  | --- ClientHello ----------> |  (Client proposes cipher suites, random)
  |                             |
  | <-- ServerHello ----------- |  (Server picks suite, sends cert & random)
  |                             |
  | <-- Certificate ------------|
  |                             |
  |     [Client verifies cert]  |
  |                             |
  | --- Pre-Master Key (Encrypted) --> |
  |                             |
  |  [Both compute session key] |
  | --- Finished (encrypted) -->|
  | <-- Finished (encrypted) ---|
  |                             |
  🔒 Encrypted communication begins

Example of TLS Handshake

Scenario: A client makes a request to https://api.example.com

1. Client (browser or HTTP client) sends a TCP connection request.
2. After TCP handshake, client sends a ClientHello to begin TLS.
3. TLS handshake negotiates a cipher suite, and verifies the server's certificate.
4. Both generate the same session key.
5. Secure HTTPS communication starts (e.g., sending a JSON payload). Why it's important in system design:

   Component	Role of TLS
   API Gateway (e.g., NGINX, Envoy)	Terminates TLS, protects downstream services
   CDN (e.g., Cloudflare)	Uses TLS to secure edge-to-origin traffic
   Microservices with mutual TLS (mTLS)	Authenticates both client and server, adds zero-trust security
   Mobile apps	Use TLS to securely communicate with backend servers

Example of TLS Handshake with Node.js

Server

const https = require("https");
const fs = require("fs");

const options = {
  key: fs.readFileSync("server.key"), // Private key
  cert: fs.readFileSync("server.crt"), // Certificate (signed by CA)
};

https
  .createServer(options, (req, res) => {
    res.writeHead(200);
    res.end("Hello TLS-secured world!");
  })
  .listen(443);

const https = require("https");

https.get("https://localhost", (res) => {
  res.on("data", (d) => {
    process.stdout.write(d);
  });
});

Under the hood, the https module performs the entire TLS handshake automatically using Node's tls module (part of OpenSSL).

TLS vs TCP

Feature	TCP	TLS
Layer	Transport Layer (L4)	Application Layer (L5–6)
Security	None	Provides encryption, authentication
Handshake	3-way	Multi-step handshake with cryptographic exchange
Used In	HTTP, FTP, DB	HTTPS, SMTPS, secure APIs, VPN