Caching

Welcome to the definitive guide on Caching for system design interviews. In high scale distributed systems, caching is the fundamental mechanism used to handle massive read traffic, eliminate database bottlenecks, and achieve low latency response times. Reading data from a traditional disk backed relational database like PostgreSQL can take 50 milliseconds or more, whereas fetching data from an in memory cache like Redis takes approximately 1 millisecond. This represents a 50x performance improvement. Databases store data on physical disk storage where every query pays the penalty of disk input output operations. Memory sits significantly closer to the central processing unit, bypassing disk bottlenecks entirely.

While caches cut latency and protect database infrastructure, they also introduce architectural challenges around data invalidation, race conditions, and system consistency. In this comprehensive guide, we walk through the core concepts of caching, explore where and how to cache, analyze cache access patterns, and master production grade strategies for handling high traffic edge cases.

The 5 Step System Design Framework

When discussing caching during a system design interview, follow this structured framework to ensure a clear, methodical, and collaborative technical evaluation:

1. Define the Requirements & Estimations: Quantify read write ratios, latency targets, and estimate total memory footprint requirements.
2. Identify Core Entities & Cache Key Design: Structure database models and establish deterministic cache key naming namespaces.
3. Design API & Cache Access Contracts: Choose appropriate cache interaction patterns based on data consistency requirements.
4. Construct High Level Design: Map out component interactions including client layers, load balancers, application servers, cache clusters, and database read replicas.
5. Deep Dives & Production Optimization: Resolve complex edge cases such as cache stampedes, hot keys, eviction policies, and cache invalidation failures.

Step 1: Requirements & Estimations

Before introducing a caching layer into an architecture, clarify the performance bottlenecks and quantify the system requirements.

Functional Requirements

• Transparent Cache Operations: Fetch requested items with minimal latency, falling back seamlessly to the persistent database upon a cache miss.
• Explicit Cache Invalidation: Provide capabilities to clear or update specific keys when underlying application state changes.
• Automatic Expiration Management: Enforce configurable TTL time to live parameters to remove stale items automatically.
• Support for Rich Data Types: Handle key value pairs, collections, lists, and geospatial structures efficiently.

Non Functional Requirements

• High Availability: Prioritize uptime over strict atomic consistency, ensuring the system functions reliably even during node additions or hardware failures.
• Low Latency Performance: Deliver read responses in under 5 milliseconds for cached items.
• Horizontal Scalability: Support transparent sharding and partitioning across distributed cache clusters to scale memory linearly.
• Graceful Degradation: Prevent cascading system failures by using circuit breakers and rate limiting if the caching tier experiences partial or total outages.

Hardware Estimations & Latency Comparison

Understanding memory and latency trade offs is essential for sizing clusters accurately:

• CPU L1 Cache Reference: 0.5 nanoseconds
• Main RAM Access: 100 nanoseconds
• Datacenter Network Roundtrip: 500 microseconds
• Redis In Memory Read: 1 millisecond
• Flash SSD Storage Read: 2 milliseconds
• Standard Magnetic Disk Seek: 10 milliseconds
• Complex Database Query: 20 to 50 milliseconds

To estimate memory footprint, consider an application with 10 million daily active users where each active user profile object consumes 1 kilobyte of memory. If we cache 100 percent of active profiles, the required raw cache memory is 10 million multiplied by 1 kilobyte, which equals 10 gigabytes of RAM. Accounting for indexing overhead and cluster replication factors, provisioning a 30 gigabyte Redis cluster split across 3 nodes guarantees high availability and operational headroom.

Step 2: Core Entities & Cache Key Design

Organizing data structures cleanly is critical for high throughput operations and avoiding key collisions.

Database Schema Representation

-- Primary database entities stored on persistent disk
CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    full_name VARCHAR(255) NOT NULL,
    profile_picture_url TEXT,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE products (
    id BIGSERIAL PRIMARY KEY,
    sku VARCHAR(64) UNIQUE NOT NULL,
    title VARCHAR(255) NOT NULL,
    price_cents BIGINT NOT NULL,
    inventory_count INT NOT NULL,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

Cache Key Naming Conventions & Data Structures

In a distributed cache tier, use colon delimited hierarchical namespaces to establish distinct key boundaries:

• User Profile String: Key user:1042:profile mapped to JSON string payloads for rapid single entity lookups.
• Product Inventory Hash: Key product:8819:details mapped to Redis Hashes with fields title, price_cents, and inventory_count to allow partial field updates without rewriting whole objects.
• Global Leaderboard Sorted Set: Key leaderboard:global mapped to Redis Sorted Sets scoring user scores for real time rankings.
• User Followers Set: Key user:1042:followers mapped to Redis Sets for constant time membership checks.

Step 3: API & Cache Access Contracts

The interface between application servers and caching systems dictates operational performance and data freshness guarantees.

1. Cache Aside Pattern (Lazy Loading)

This is the standard caching pattern for read heavy applications. Application code orchestrates reads and writes between the cache and the database directly.

GET /api/v1/users/1042

Read Execution Flow:

1. Application receives client request for user 1042.
2. Application queries cache using key user:1042:profile.
3. If cache hit occurs, application returns cached payload immediately.
4. If cache miss occurs, application queries persistent database, stores result in cache with a 15 minute TTL, and returns payload to client.

PUT /api/v1/users/1042
Content-Type: application/json

{
  "full_name": "Jane Doe"
}

Write Execution Flow:

1. Application updates underlying row in PostgreSQL database.
2. Application issues invalidation command to delete key user:1042:profile from cache.

2. Write Through Pattern

In write through caching, application code writes strictly to the cache service. The cache framework synchronously persists updates to the database before returning success to the client application. This pattern guarantees that cache and database remain perfectly synchronized, though write operations incur higher latency.

3. Write Behind Pattern (Write Back)

In write behind caching, updates write directly to the cache tier and acknowledge completion to the client immediately. A background worker process asynchronously flushes aggregated writes to the relational database in batches. This pattern delivers ultra high write throughput, but risks minor data loss if cache nodes fail before flushing queued writes to disk.

4. Read Through Pattern

In read through caching, the application treats the cache layer as a transparent proxy. On a cache miss, the cache infrastructure automatically handles fetching data from the database, storing it internally, and delivering it to the caller. Content Delivery Networks CDNs operate primarily as read through proxies.

Step 4: High Level Design

The high level architecture illustrates how request routing, caching layers, and database clusters interact across different tiers of the system infrastructure.

Architecture Diagram Generation Prompt: A comprehensive system architecture block diagram showing client devices connecting through an Edge CDN layer to an API Gateway and Load Balancers cluster. The application servers connect to a multi node distributed Redis cluster acting as an external cache tier, with an in process local memory cache configured inside application nodes for ultra hot keys. Behind the application servers lies a primary PostgreSQL relational database with automated asynchronous read replicas to handle persistent storage and fallback traffic.

System Architecture Operational Flow

• Edge CDN Layer: Intercepts inbound static media requests and cached public API responses at edge locations worldwide, reducing latency from 250 milliseconds down to under 30 milliseconds.
• API Gateway & Load Balancers: Distribute client traffic evenly across stateless application instances while handling authentication and TLS termination.
• Application Server Tier: Executes business logic and implements dual layer caching. Frequently accessed reference data resides in local in process memory, while shared user state queries hit the distributed external Redis cluster.
• External Distributed Cache Tier: A cluster of Redis nodes operating with primary secondary replication and automatic sharding, serving low latency reads to shield the database.
• Persistent Database Tier: Primary database handles all transactional write operations and streams WAL logs asynchronously to dedicated read replicas.

Step 5: Deep Dives & Production Case Studies

Navigating complex production edge cases separates novice system architects from senior engineering leaders.

1. Cache Eviction Policies

When memory bounds are reached, cache engines execute configured eviction algorithms to reclaim capacity:

• Least Recently Used LRU: Tracks access recency and evicts keys that have not been requested for the longest duration. This is the recommended default for general web workloads.
• Least Frequently Used LFU: Tracks request frequency counts per key and evicts items with the lowest hit counts. Ideal for datasets with clear long term popularity trends.
• First In First Out FIFO: Evicts keys strictly based on insertion age regardless of access patterns. Rarely used in modern production systems due to suboptimal cache hit rates.
• Time To Live TTL Expiration: Sets strict expiration timestamps on keys. Often combined with LRU policies to guarantee that stale entries eventually expire even if memory is abundant.

2. Resolving Cache Stampedes (Thundering Herd)

A cache stampede occurs when a high traffic key with a heavy database generation cost expires. Thousands of concurrent client requests experience a simultaneous cache miss and crush the underlying database.

Mitigation Strategies:

• Request Coalescing Singleflight: Lock execution so that only the first thread executes the database query while subsequent concurrent threads wait and share the resulting cache payload.
• Probabilistic Early Expiration: As keys approach their expiration threshold, algorithms randomly trigger background refreshes based on request velocity, preventing key hard drops.
• Proactive Cache Warming: Scheduled background jobs compute and refresh critical system keys before TTL expiration.

3. Solving Hot Key Bottlenecks

In massive scale applications, single keys can experience extreme traffic spikes. For instance, viral celebrity profiles or breaking news alerts can overwhelm an individual Redis cluster node.

Mitigation Strategies:

• Key Replication & Sharding: Append random suffix counters to keys (such as news:viral_article:shard_1 through shard_10) to distribute read operations across multiple Redis nodes.
• In Process Local Fallback: Cache extreme hot keys directly in application server memory for short durations (such as 5 seconds), completely shielding the external cache layer from spikes.

4. Real World Production Case Studies

• Strava Mobile Offline Synchronization: Strava caches athlete run metrics directly on physical mobile devices during activity recording. The app relies on client side local storage while network connections are absent and executes background sync operations once connectivity returns.
• Twitter / X Profile & Timeline Caching: Twitter relies heavily on distributed Redis clusters using cache aside patterns to serve user timelines instantly. Extremely popular accounts utilize key sharding and local in process caching to prevent single node memory bottlenecks.
• Netflix CDN Edge Video Delivery: Netflix deployed Open Connect, a proprietary custom Content Delivery Network. By caching video assets at edge locations directly inside Internet Service Provider networks globally, Netflix bypasses internet transit bottlenecks and delivers high definition video streaming smoothly.

Mastering caching principles enables engineers to build resilient, ultra low latency distributed systems capable of scaling seamlessly to millions of concurrent users.