Imagine hiring a software engineer. You wouldn’t just ask them to recite textbook definitions - you’d give them real bugs to fix, systems to design, and deadlines to hit. AI benchmarks work the same way. Each one is a different job interview for a machine mind, testing whether it can actually do the work - not just talk about it.

This guide breaks down every major AI benchmark in plain language: what it tests, why it matters, what happens when AI aces it, and what keeps researchers up at night.

1. SWE-bench Verified - “Can You Actually Fix Real Bugs?”

Picture this: it’s 2 a.m., your production server is down, and the bug is buried somewhere in a 50,000-line Django codebase. You need someone who can read the issue report, navigate the repo, understand the architecture, write the patch, and make sure the tests pass - all without hand-holding.

That’s SWE-bench.

What It Actually Tests

SWE-bench pulls real GitHub issues from popular open-source projects (Django, Flask, scikit-learn, sympy, etc.) and asks the AI to produce a working code patch. The “Verified” variant means every problem has been human-reviewed to confirm it has a clear, unambiguous solution.

• The AI gets: an issue description and the full repository.

• The AI must: produce a code diff that fixes the issue.

• Success is measured by: whether the existing test suite passes after applying the patch.

Current Score: 93.9%

This means the AI can now fix 94 out of 100 real software bugs from production open-source projects. A year ago, the best models were stuck around 30%.

What Happens When This Hits 100%

Automated bug triage at scale - Companies could route incoming GitHub issues directly to an AI agent that fixes and opens a PR within minutes.

• 10x reduction in maintenance cost - The bulk of any engineering team’s time is spent on bug fixes and incremental changes, not greenfield features.

• Open source acceleration - Projects with thousands of stale issues (like CPython, Linux) could be systematically cleaned up.

Example: Imagine every good-first-issue on GitHub getting an automated fix PR within 5 minutes of being filed. New contributors could focus on design and architecture instead of hunting typos and edge cases.

Risks

• “Looks correct, isn’t correct” :A patch can pass tests but introduce subtle regressions. The test suite is only as good as its coverage.

• Benchmark gaming :Models trained specifically on SWE-bench repos might memorize patterns rather than truly understanding code.

• Job displacement anxiety :Junior developers often build their skills through bug-fix tickets. If AI handles those, how do juniors learn?

2. SWE-bench Pro - “Can You Handle the Hard Stuff?”

SWE-bench Verified was the appetizer. SWE-bench Pro is the main course and it’s trying to choke you.

While “Verified” tests single-file fixes in Python repos, Pro throws in multi-language codebases, multi-file edits, complex dependency chains, and problems that require understanding the architecture of a system, not just its syntax.

Current Score: 77.8%

Still very high, but the 16-point gap from Verified tells you something important: understanding architecture is harder than fixing syntax.

What This Means In Practice

If this reaches 95%+:

• AI could handle complex refactoring tasks (“migrate this codebase from REST to GraphQL”)

• Pull request reviews could become fully automated not just linting, but semantic code review

• Mid-level engineering tasks start becoming AI-automatable

Example: A startup with 3 engineers could ship at the velocity of a 15-person team, with AI agents handling the bulk of feature branch work while humans focus on product direction.

3. USAMO 2026 - “Can You Think Like a Math Olympiad Champion?”

The USA Mathematical Olympiad (USAMO) isn’t your school exam. There are no multiple-choice questions. No calculators. Just six brutally hard proof-based problems spread across two 4.5-hour sessions. The best high school mathematicians in America score maybe 25 out of 42 points on a good day.

Now an AI scores 97.6%.

What It Actually Tests

USAMO problems require:

• Creative insight - You can’t brute-force a proof. You need to “see” the elegant path.

• Rigorous logic - Every step must be watertight. Hand-waving gets you zero points.

• Multi-step reasoning - Solutions often span 2–3 pages of dense mathematical argument.

Why This Is Mind-Blowing

This isn’t about calculating 2 + 2. USAMO tests the kind of abstract reasoning that was supposed to be uniquely human - the ability to stare at a problem, have an “aha!” moment, and construct an argument that didn’t exist before.

Example problem flavor: “Prove that for all positive integers n, there exists a set of n points in the plane, no three collinear, such that all mutual distances are irrational, but every triangle formed has rational area.”

A human proving this needs geometric intuition, number theory knowledge, and creative construction. The AI now does this better than 99% of trained mathematicians.

Benefits

• Automated mathematical research - AI could generate and verify new theorems, accelerating fields like number theory, topology, and combinatorics.

• Personalized math tutoring - An AI that can prove can also explain - walking students through proofs step by step, identifying exactly where their reasoning breaks down.

• Engineering applications - Better formal verification of safety-critical systems (bridges, aircraft, nuclear reactors) through automated mathematical proof checking.

Risks

• The “creativity” question :Is the AI truly being creative, or has it absorbed enough proof patterns to generate novel-looking combinations? The philosophical debate rages on.

• Academic integrity :Math competitions and proof-based courses face a fundamental challenge when AI can outperform every student.

4. GPQA Diamond - “Are You Smarter Than a PhD?”

Here’s a humbling experiment: Take 198 multiple-choice questions in biology, chemistry, and physics. Give them to smart, educated non-experts with unlimited time and full internet access. They score about 34%. Give them to PhD domain experts. They score about 65–70%.

Now give them to AI. It scores 94.6%.

The AI is outperforming the humans who wrote the questions.

What Makes GPQA Diamond Special

The “Google-Proof” in the name is the key: these questions are deliberately crafted so you can’t just look up the answer. They require:

• Synthesizing knowledge across sub-fields (e.g., applying quantum mechanics to a biology question)

• Multi-step reasoning through complex chains of logic

• Understanding at the intuition level, not just the definition level

Example question flavor: “A researcher observes anomalous spin-orbit coupling in a novel 2D material. Given the following band structure data and crystal symmetry constraints, which of these four mechanisms best explains the observed splitting pattern?”

You can’t Google that. You either understand condensed matter physics or you don’t.

What 94.6% Means

Practical Benefits:

• AI-assisted diagnosis - An AI that reasons at PhD level in biology and chemistry could serve as a “second opinion” for complex medical diagnoses, especially in regions with doctor shortages.

• Accelerated drug discovery - Multi-disciplinary reasoning (combining chemistry, biology, and physics) is the bottleneck in pharmaceutical research. AI could propose novel drug candidates by connecting dots that siloed human teams miss.

• Graduate education - PhD students could have a study partner that genuinely understands their domain at an expert level.

Risks

• Confident but wrong : At 94.6%, the AI still gets 5.4% wrong. In medicine or materials science, that 5% could be catastrophic if humans over-trust the model.

• Deskilling : If researchers rely on AI to do their reasoning, do they lose the ability to think critically themselves?

5. Humanity’s Last Exam - “The Hardest Test Ever Created”

In late 2024, a global consortium of experts set out to create the hardest exam ever written. The rules were brutal:

1. Domain experts from every field - math, literature, history, law, medicine, obscure linguistics - submitted their hardest questions.

2. If any existing AI model could answer a question, it was thrown out.

3. The remaining 2,500 questions were supposed to be the last frontier - the intellectual Berlin Wall that machines couldn’t breach.

When it launched, the best AI scored 8%. Researchers celebrated. “This will keep them busy for years,” they said.

It took 16 months. We’re now at 64.7%.

What It Tests

Everything. Literally everything academic humans know:

• Advanced mathematics and logic

• Obscure historical facts and literary analysis

• Expert-level scientific reasoning

• Philosophical arguments

• Multilingual comprehension

• Cross-domain synthesis (questions that require expertise in TWO fields simultaneously)

The 64.7% Score (With Tools)

“With tools” means the AI was allowed to write and run code, search for information, and use calculators. Without tools, it still scored 56.8%.

Why this is significant: The exam was designed to be impossible for AI. Reaching 64.7% is like having someone break a world record that everyone said was physically unattainable.

Benefits

• True general intelligence indicator - Unlike narrow benchmarks, HLE tests breadth. A high score here means the AI is genuinely knowledgeable across all human domains.

• Universal research assistant - An AI that can answer questions across every discipline becomes the ultimate interdisciplinary tool.

Risks

• The “last exam” problem : If AI conquers this too, what’s the next benchmark? At what point do we run out of ways to measure whether AI is smarter than us?

• False sense of superintelligence : 64.7% is impressive but still means the AI is wrong on a third of the hardest questions. Misunderstanding this as “AI knows everything” creates dangerous overconfidence.

6. CyberGym - “Can You Hack Like a Professional?”

Cybersecurity is a cat-and-mouse game. Defenders build walls; attackers find cracks. CyberGym puts AI on both sides of that war.

The benchmark includes real-world offensive and defensive tasks:

• Finding vulnerabilities in code

• Writing exploits

• Detecting intrusions

• Reproducing known CVEs (Common Vulnerabilities and Exposures)

Current Score: 83.1% (vs 66.6% for previous generation)

This 16.5-point jump means the new AI can:

• Find and exploit vulnerabilities that the previous model completely missed

• Chain together multi-step attacks (e.g., privilege escalation → lateral movement → data exfiltration)

• Understand defense mechanisms well enough to bypass them

Practical Benefits

• Automated penetration testing - Companies could run continuous, AI-driven security audits instead of expensive annual pen-tests.

• Bug bounty automation - AI could scan open-source code for vulnerabilities before attackers find them.

• National cybersecurity - Governments could deploy AI defenders that match the capabilities of the best human attackers.

Risks

This is where the scoreboard gets dark.

• Offensive capabilities in the wrong hands : An AI that can find and exploit vulnerabilities at 83.1% accuracy is a weapon. If this technology is open-sourced or leaked, it dramatically lowers the barrier for cyberattacks.

• Speed advantage : A human pen-tester might find a vulnerability in hours. An AI can find hundreds in minutes. The attacker-defender asymmetry gets worse.

7. Firefox Exploit Writing - “Can You Write a Zero-Day?”

This is the benchmark that made Anthropic not release their model to the public.

In red-team testing against Firefox 147 (a fully patched, modern web browser), Claude Mythos Preview successfully developed working JavaScript shell exploits 181 times. The previous model? Twice.

Let that sink in. A 90x improvement in the ability to autonomously find and exploit zero-day vulnerabilities in production software used by hundreds of millions of people.

What This Actually Means

A “zero-day” is a vulnerability that the software vendor doesn’t know about - so there are zero days of advance warning. They are the nuclear weapons of cybersecurity, and historically, only elite nation-state hacking groups and a handful of expert researchers could develop them.

Now an AI can do it 181 times in a test session.

Practical Benefits (if properly controlled)

• Proactive vulnerability discovery - If the good guys use this first, every major browser, OS, and framework could be hardened before attackers find the holes.

• The “immune system” model - Think of it like AI white blood cells, constantly probing your own software for weaknesses and alerting developers.

Risks (and why this model is restricted)

• This is why Claude Mythos Preview is not publicly available. Anthropic restricted access to select security partners under “Project Glasswing” an initiative focused on securing critical software.

• Proliferation risk : If this capability leaks into open-source models, the entire internet’s security model changes overnight.

• Autonomous attack potential : An AI that can write browser exploits could theoretically be chained with other AI capabilities to conduct end-to-end cyberattacks without human involvement.

This is no longer theoretical. This benchmark represents the first concrete evidence that AI has crossed the threshold from “cybersecurity tool” to “cybersecurity weapon.”

8. Cybench CTF Challenges - “Can You Win a Hacking Competition?”

Capture The Flag (CTF) competitions are the Olympics of cybersecurity. Teams of elite hackers compete to solve challenges across categories like cryptography, reverse engineering, binary exploitation, web security, and forensics. Each challenge has a hidden “flag” (a secret string) that you can only find by successfully exploiting a vulnerability or solving a puzzle.

Cybench packages professional-level CTF challenges into a standardized benchmark for AI agents.

What It Tests

Unlike CyberGym (which focuses on broader offensive/defensive tasks), Cybench is specifically about CTF-style puzzle-solving:

• Reverse engineering - Disassemble a binary, figure out what it does, extract the secret

• Cryptography - Break flawed encryption schemes

• Web exploitation - Find SQL injection, XSS, or authentication bypasses in a running web app

• Binary exploitation - Write buffer overflow exploits, ROP chains, format string attacks

• Forensics - Recover hidden data from memory dumps, network captures, or corrupted files

Current Performance

Frontier models have been rapidly improving on Cybench, with top agents now capable of solving tasks that would take skilled human CTF players hours. While no model has achieved 100% on the full benchmark, some specialized agent configurations have demonstrated near-perfect performance on curated subsets of challenges, particularly in web exploitation and cryptography categories.

Practical Benefits

• Training the next generation - AI CTF solvers can generate unlimited practice problems and walkthroughs for cybersecurity students

• Automated security audits - The same skills that crack CTF challenges (SQL injection, XSS, auth bypass) are exactly what real-world pen-testers look for

• Vulnerability research - An AI that can reverse-engineer binaries and break crypto could find flaws in real software before attackers do

Risks

• Lowered barrier to entry: CTF skills traditionally required years of training. An AI that solves them instantly could give script kiddies the power of expert hackers.

• Dual-use by design : Every CTF skill maps directly to a real-world attack technique. There’s no “safe” version of binary exploitation.

9. Terminal-Bench 2.0 - “Can You Run a Server?”

Forget writing code - can the AI actually operate an entire system? Terminal-Bench 2.0 drops the AI into a sandboxed Linux container with a task like:

• “Debug why this Docker container keeps crashing”

• “Migrate this legacy Python 2 codebase to Python 3”

• “Set up a CI/CD pipeline for this project”

• “Find and fix the memory leak in this Node.js application”

No GUI. No hand-holding. Just a terminal prompt and a job to do.

Current Score: 82.0%

This means the AI successfully completes 82% of real-world DevOps and system administration tasks by navigating file systems, reading logs, editing configs, running commands, and debugging issues - all through a terminal.

Practical Benefits

• AI DevOps engineers - On-call rotations could be supplemented (or replaced) by AI agents that diagnose and fix infrastructure issues at 3 a.m.

• Automated migration projects - The dreaded “upgrade from v2 to v3” project could be handled by an agent that reads docs, updates code, and validates the result.

• Small team force multiplier - A startup founder who can’t afford a dedicated SRE gets one for the cost of an API call.

Risks

• Root access concerns : An AI operating with terminal access has enormous power. A mistyped rm -rf / or misconfigured firewall rule can be catastrophic.

• Blind trust in automation : When the AI “says” it fixed the issue, how do you verify? The 18% failure rate means nearly 1 in 5 tasks isn’t completed correctly.

If you’re curious about Agentic AI but don’t know where to start, I’ve broken it down in the simplest possible way for beginners , checkout here

10. OSWorld-Verified - “Can You Use a Computer Like a Human?”

OSWorld takes it a step further than Terminal-Bench. Instead of just a terminal, the AI gets a full desktop environment - Ubuntu, Windows, or macOS - and must complete tasks by clicking buttons, navigating menus, filling out forms, and switching between applications.

Tasks like:

• “Open the spreadsheet, sort column C by date, create a pivot table, and email the results to john@example.com”

• “Install VS Code, configure the Python extension, and set up a virtual environment for this project”

• “Find the file called ‘Q3_report.pdf’ somewhere on this computer and upload it to Google Drive”

Current Score: 79.6%

The AI can now use a computer like a human user about 80% of the time - navigating GUIs, understanding visual layouts, and performing multi-step workflows across applications.

Practical Benefits

• True digital assistants - Not chatbots, but agents that operate your computer for you. “Schedule a meeting, book the conference room, and send the calendar invite” - done.

• Accessibility revolution - Elderly or disabled users who struggle with complex interfaces could delegate tasks to an AI that operates the computer on their behalf.

• Business process automation - RPA (Robotic Process Automation) on steroids. Instead of brittle scripts that break when a button moves, AI agents that understand the interface.

Risks

• Privacy and access : An AI that can operate your full desktop can also see your emails, passwords, and private files.

• Shadow IT : If employees start using AI agents to automate their work, IT departments lose visibility into what’s happening on company machines.

11. MMMLU - “Do You Understand the Whole World?”

MMMLU (Multilingual Massive Multitask Language Understanding) takes the classic MMLU knowledge test - thousands of questions across 57 subjects - and runs it in dozens of languages. It’s not enough to know the answer in English; can you explain quantum physics in Hindi? Debate philosophy in Japanese? Discuss constitutional law in Portuguese?

Current Score: 92.7%

The AI demonstrates near-expert-level knowledge across most academic subjects and in most major world languages.

Practical Benefits

• Global education access - A student in rural Indonesia gets the same quality AI tutor as a student at MIT, in their own language.

• Multilingual governance - AI systems that can reason in the local language help governments serve diverse populations.

• Cross-cultural research - Researchers can query AI in their native language and get domain-expert-level answers without English as a bottleneck.

Risks

• Cultural bias - The AI might “know” many languages but still reason from a Western-centric perspective baked into its training data.

• Low-resource language gap - While scores are high for major languages (English, Chinese, Spanish), performance drops significantly for languages with less training data (Swahili, Quechua, Welsh).

The Bigger Picture: What All These Numbers Mean Together

Look at the scoreboard again. Every single benchmark shows massive year-over-year improvement. But the combination is what matters:

The Timeline

Think of each benchmark as a different subject in school:

• 2023: AI was a mediocre high school student - decent at multiple choice, terrible at essays and proofs.

• 2024: AI graduated to college level - could write code, solve standard problems, and hold conversations about most topics.

• 2025: AI became a grad student - could tackle research-level problems, write production code, and reason about complex systems.

• 2026: AI is now a world-class expert in coding, math, science, and cybersecurity - simultaneously.

These benchmarks aren’t just academic games. Each one represents a real-world capability that, when mastered, changes what’s possible. We’re watching AI cross threshold after threshold, and the speed of crossing is accelerating. The question is no longer “Will AI get there?” it’s “Are we ready when it does?”

Share

Use this as your high-yield interview revision page. It is not meant to cover every topic. It is meant to cover the things that repeat across DSA, System Design, LLD, and interview rounds often enough to matter.

Pair it with the main roadmaps, then use the lists below as your last-mile revision checklist. LLD Roadmap , HLD Roadmap , DSA Roadmap

What to prioritize first

1. DSA first - Prioritize this for internship, fresher, and SDE 1 style interviews where coding rounds dominate.

2. System Design next - Shift here for SDE 2+, backend, platform, API, and distributed systems roles where scale and tradeoffs matter.

3. LLD to close gaps - Practice this heavily if the company likes machine coding, object modeling, design patterns, or real-world code structure discussions.

Target Outcome

If you can solve most of the DSA list below, explain major System Design tradeoffs clearly, and walk through 4–5 LLD problems without drifting into vague class dumping, you are already in strong shape for a large number of interviews.

DSA: the patterns that show up again and again

• Arrays and hashing

• Two pointers

• Sliding window

• Binary search

• Stack and monotonic stack

• Queue and heap

• Linked list

• Trees and graphs

• Backtracking

• Trie

• Dynamic programming

• Greedy

25 must-do DSA questions

Arrays, hashing, and sliding window

1. Find Pair With Sum : Baseline hashing pattern. You should be able to compare brute force, sorting, and hash map approaches cleanly.

2. Duplicates Within K Distance : Simple but useful for learning set-based windows and sliding constraints.

3. Rotate Image : Classic in-place matrix manipulation problem that tests index discipline.

4. 3 Sum to Zero : Sorting, duplicate handling, and two-pointer reasoning in one question.

5. Find All Anagrams : Strong sliding-window frequency-map question that appears in many variants.

Binary search, greedy, and interval thinking

1. Minimum Window Substring : One of the most important sliding window patterns to master deeply.

2. Find First and Last Position : Teaches clean boundary-based binary search instead of just “find target”.

3. Search Rotated Array II : Binary search with ambiguous duplicate cases and branching discipline.

4. Split Array Largest Sum : Important example of binary search on answer space.

5. Jump Game II : Greedy range expansion question with very interview-friendly intuition.

Stack, queue, and heap

1. Balanced Brackets : Basic stack correctness, but still useful for edge cases and clean thinking.

2. Implement Queue Using Stacks : Tests amortized analysis and data structure simulation.

3. Largest Rectangle in Histogram : The classic monotonic stack problem.

4. Task Scheduler: Greedy counting and cooldown reasoning with useful follow-up discussions.

5. Merge K Sorted Lists : Heap merge pattern that generalizes to many stream-style problems.

Linked list, trees, and graphs

1. Intersection of Two Linked Lists : Small problem, but excellent for pointer reasoning and explanation quality.

2. Sort List : Merge sort on linked lists. Common and very reusable.

3. Diameter of Binary Tree : Bottom-up tree recursion and “answer through child state” thinking.

4. Construct Tree From Traversals : Recursion plus indexing map. Great for testing tree reconstruction logic.

5. Count Islands : Core DFS/BFS connected-component foundation problem.

Backtracking, trie, DP, and DSU

1. Find Redundant Connection : Union-find basics and cycle detection in an elegant format.

2. Combination Sum : Backtracking template, branching choices, and pruning.

3. Add and Search Words : Trie plus wildcard branching. Good for pattern-matching structures.

4. Count Ways to Climb : The cleanest entry point into one-dimensional dynamic programming.

5. Longest Common Subsequence : One of the essential two-dimensional DP problems.

DSA revision rule

• State the brute-force solution first. Interviewers want to know that you can reason from baseline to optimized.

• Explain why the optimized structure works. Do not just say “use a map” or “use DP” without the invariant.

• Call out time and space complexity clearly. Say it before the interviewer has to ask.

• Walk through one edge case. Empty input, duplicates, overflow, off-by-one, or single-element cases usually reveal whether you really understand the solution.

• Practice verbalizing while coding. Silent solving does not prepare you for real rounds.

System Design: what actually matters in interviews

• Clarify requirements. Separate functional requirements from scale, latency, consistency, and reliability expectations.

• Estimate rough scale. Users, QPS, storage, peak traffic, and read/write ratio shape the design.

• Define APIs and data model. This forces your design to become concrete early.

• Draw high-level components. Client, gateway, services, cache, database, async workers, and storage.

• Deep dive into the real bottleneck. Pick the place where the design gets interesting instead of staying generic.

• Handle failures and observability. Retries, DLQs, fallback behavior, metrics, logs, tracing, and alerts matter.

• End with tradeoffs. Good design interviews end with constraints, not false certainty.

High-yield System Design topics

• Back-of-the-Envelope : EstimationPractice quick reasoning about QPS, storage, bandwidth, and latency budgets.

• CAP Theorem : Foundational language for distributed tradeoffs.

• Consistency Models : Know strong, eventual, causal, and when each is acceptable.

• L4 vs L7 Load Balancing : A very common scaling and routing discussion.

• Cache Invalidation Strategies : One of the most practical topics in real systems.

• Database Replication : Leader-replica basics, lag, and failover thinking.

• Database Sharding : When single-node storage stops being enough.

• Message Brokers : Queues, fan-out, decoupling, and async workflow design.

• Retry and Backoff Strategies : Critical for resilience and client behavior under failure.

• Dead Letter Queues : What happens when retries keep failing.

• Distributed Tracing : Observability is part of good design, not a side note.

• Exactly-Once Payment Processing : Idempotency and correctness under retries.

• Multi-Region Design : Latency, failover, routing, and cross-region data decisions.

• Disaster Recovery : RPO, RTO, backups, restore testing, and operational realism.

Classic System Design problems you should be ready for

• Design a URL Shortener : Hashing, key generation, storage, redirects, hot keys, and analytics.

• Design a News Feed : Fan-out, ranking, caching, and read/write tradeoffs.

• Design a Notification System : Retries, templates, preferences, rate limits, and delivery guarantees.

• Design a Rate Limiter : Fixed window, sliding window, counters, and distributed fairness.

• Design a Ticket Booking System : Seat holds, locking, contention, timeout expiry, and payment correctness.

Complete list of 50+ HLD questions

What interviewers want to hear in System Design

• Where the data lives. Not just “use a database,” but which store holds which truth.

• Where the bottleneck appears first. Cache hot keys, database contention, queue lag, fan-out pressure, or storage throughput.

• How reads and writes scale differently. Caches, replicas, queues, batching, or partitioning should come up naturally.

• How duplicate work is prevented. Idempotency, dedupe keys, exactly-once semantics, and safe retries matter.

• How failures degrade. What happens when a dependency is slow, down, or partially failing.

• What tradeoff you are making on purpose. Strong answers are explicit about the cost of simplicity, latency, consistency, or freshness.

Low Level Design: the parts that separate average answers from strong ones

The goal is not to create many classes. The goal is to model the domain cleanly, keep invariants close to where they belong, and leave space for future requirements without collapsing into a God object.

High-yield LLD topics

• SOLID in Real World : Know where each principle helps and where over-application makes code worse.

• Composition vs Inheritance : Composition should usually be your default.

• Encapsulation : Keep invariants inside objects instead of scattering checks.

• Abstraction : Expose what matters, hide what changes.

• Dependency Injection: Cleaner wiring, easier tests, lower coupling.

• Dependency Inversion : Depend on contracts instead of concrete implementations.

• Repository Pattern : Separate persistence details from domain behavior.

• Service Layer : Coordinate workflows without bloating controllers or entities.

• Thread Safety : Essential whenever shared mutable state exists.

• Synchronization Primitives : Know locks, semaphores, and coordination basics.

• Strategy Pattern : Great when behavior varies cleanly behind an interface.

• Observer Pattern : Useful for event-driven updates and decoupled notifications.

• Factory Method : Hide creation complexity and keep callers clean.

• Decorator Pattern : Extend behavior without subclass explosion.

• State Pattern : Perfect when behavior changes with lifecycle stages.

• Clean Architecture : Boundaries, use cases, and dependency direction.

• Hexagonal Architecture: Ports-and-adapters thinking for clean change boundaries.

• Unit Testing : Interview code should be testable even if you do not write every test.

• Long Methods and God Classes : Recognize the smell before you produce it.

A strong LLD answer usually includes

• Core entities and responsibilities. Every important domain object should have a reason to exist.

• Interfaces where behavior varies. Do not hardwire future variation into conditionals if an abstraction is cleaner.

• A clear workflow for the main use case. Walk through the request path from input to state change.

• A state model when lifecycle matters. Booking, payment, order, and approval flows usually benefit from explicit state thinking.

• Validation and invariant ownership. Decide which object is responsible for enforcing correctness.

• Concurrency handling. If two users can update shared state, you need a strategy for correctness.

• Extension points. Think about how the design evolves when a new rule or variant appears.

• Tests you would write first. This reveals whether the design is actually usable.

Classic LLD problems you should practice

• Design Parking Lot : Entities, spot assignment, ticketing, pricing, and entry/exit flow.

• Design LRU Cache : Hash map + doubly linked list with strict O(1) operations.

• Design Splitwise : Domain modeling, expense strategies, and simplification choices.

• Design Elevator : Scheduling, state transitions, and directional logic.

• Design Vending Machine : Inventory, payment, state handling, and rollback cases.

• Design ATM : Workflow modeling, validation, security, and state machine thinking.

Complete list of 35+ LLD questions

If you can explain the topics on this page clearly and solve most of the linked DSA questions without major hints, you already cover a large chunk of what many interviews expect.

Every engineer collects scars. The kind that don’t show up in your resume but live rent-free in your head every time you write a similar piece of code.

These nine lessons aren’t theoretical. Each one was learned inside a war room, a Slack thread during outage, or a post-mortem doc written in collective shame. If any of these sounds painfully familiar - you’re in good company.

Let’s get into them - with real examples.

The Coupling Rule

1) If two modules talk too much → merge them

Assume at one company, you had a billing module and a pricing module.

In theory, they were separate concerns. Clean architecture, nice diagrams, everyone clapped in the design meeting.

In reality? Every billing call needed pricing data. Every pricing change required a billing update. They were calling each other constantly - like a couple fighting but refusing to break up.

What life looked like every sprint

• Pricing team deploys → billing breaks

• Billing team deploys → pricing edge cases surface

• Both teams stop and blame the interface contract

• Everyone writes more tests. Nothing fundamentally changes.

One afternoon the team sat in a conference room and said the quiet part out loud: “These aren’t two modules. They’re two halves of one responsibility.”

We merged them into a single service. Now: fewer deploy issues, simpler tests, no circular dependencies, cleaner ownership.

Warning sign: If you're maintaining a shared interface document just so two modules don't break each other - the abstraction is lying to you.

Lesson: Chatter = coupling. If two parts can’t live without each other, don’t force the separation. Merge is not defeat - it’s clarity.

The Silence Rule

2) If they talk too little → introduce an event

The opposite problem showed up in another system.

The User Profile Service updated a user’s email address. Quietly. No ceremony, no announcements.

The Notification Service? It never heard a thing.

What users experienced

• Changed email → no verification email sent to new address

• No security alert about the email change

• Old email (possibly compromised) no longer gets any notifications

• Support tickets: “I never got any emails after updating my address”

You dug in and found the two modules had zero connection. No API call, no event, nothing. They just lived in separate repos doing their own thing, blissfully unaware of each other.

“The bug isn’t in either module - the bug is in the gap between them.”

You fixed it by publishing a simple event: user.email_updated.

Now the Notification Service subscribed and sent a verification. The Auth Service listened and invalidated sessions. The Audit Log captured the change. Everything downstream became consistent - without any module knowing about the others.

Lesson: Silence creates invisible bugs. If modules need awareness but not coupling - events fill the gap. The event bus is not overhead, it's connective tissue.

The Velocity Rule

3) If your logic changes often → use a rules engine

Assume you had a fraud detection system. Classic stuff - block suspicious transactions, flag risky merchants, apply region-specific policies.

Week 1: the logic fit in a single function. Clean, readable, nice.

Week 6: the product team wanted a new exception for enterprise clients.

Week 8: a region-specific override.

Week 10: a time-of-day condition.

By week 16, the function had 11 nested if blocks and nobody could read it without a whiteboard session.

The quarterly product meeting, every quarter

• PM: “Small change - just add an exception for users over $50k spend”

• Dev: “That’s a deploy. Full QA cycle. 2 weeks.”

• PM: “For one condition?”

• Dev: “...yes.”

In this case you had to switched to a rules engine. Business analysts wrote YAML-based rules, you loaded them dynamically at runtime, and 90% of changes stopped requiring a deploy entirely. QA tested rule files, not application builds.

Lesson: Fast-changing logic doesn’t belong in code. Put it in data. Interpret it at runtime. Your future self will thank you at the next product planning meeting.

The Invisibility Rule

4) If your data changes often → use DB triggers carefully

Now we tracked customer loyalty points. Every transaction updated a running balance - inserts, updates, refunds, all of it.

Instead of recalculating the balance on every read, a clever dev set up a database trigger. It updated a denormalized points_balance column automatically on every transaction insert. Elegant. Fast. Very clever.

It worked beautifully… for three months.

Then someone added a business rule inside the trigger (a rule that should have lived in the application layer). Then another person added logging to the trigger. Then a third added a conditional.

"Why did this row update?" - "Oh, the trigger ran."
"Why didn't staging behave the same?" - "Different trigger version."
"Where is the trigger defined?" - silence.

Triggers are invisible code. They don’t show up in git blame. They don’t appear in your IDE search. Most devs don’t know they exist until something breaks in a way that makes no sense.

Good trigger use cases: Maintaining a last_updated_at timestamp. Enforcing a referential invariant. Logging raw inserts to an audit table. These are bookkeeping jobs - not business logic.

Lesson: Triggers are powerful - but invisible. Use them for bookkeeping and invariants, never for business rules. What lives in the DB, stays in the DB - and haunts future engineers.

The Cold Start Rule

5) If a cache miss hurts → precompute

If you have to build an analytics dashboard. Revenue per segment, top products, user cohort breakdowns. The kind that executives open first thing Monday morning, click frantically, and then send a message to engineering because it’s slow.

You cached it. But on a cache miss - a new day, a freshly deployed service, a cache eviction - the system had to recalculate everything from scratch across millions of rows. Every cache miss caused a CPU spike that slowed down every other service on the same host.

Monday 9:03 AM, every week

• CFO opens dashboard → cold cache → 4-second spinner

• Clicks “Refresh” (mistake) → triggers the calculation again

• CPU spikes → API latency rises across the board

• On-call engineer gets paged everytime

The fix wasn’t more Redis tuning. The fix was removing the heavy computation from the request path entirely. You have to run a background job every hour that precomputed all summaries and stored them. The dashboard read from a summary_snapshots table-— always pre-warmed, always fast.

Dashboard load time: 4 seconds → 100ms. The CFO stopped messaging engineering.

Lesson: If the fallback computation is expensive, move it out of the request path entirely. Caching defers work - precomputing eliminates it.

The Truth Rule

6) If staleness hurts → invalidate aggressively

E-commerce inventory is the opposite of the dashboard problem. Here, stale data doesn’t just slow things down - it breaks trust with the customer.

Your inventory cache had a 30-second TTL. Seemed reasonable. For most data, it is.

But inventory is not “most data.” Ten seconds of a stale cache meant a customer could see “Only 1 left!”, add the item to their cart, go through checkout, enter payment details, and then hit a hard wall: “Sorry, this item is no longer available.”

"We're a technology company losing revenue because of a 30-second cache."
- Someone's post-mortem, probably.

You tore out the TTL-based invalidation and replaced it with event-driven invalidation. Every purchase, restock, and cancellation published an inventory.changed event. The cache entry died the moment the stock did.

Caches are not slower databases. They are temporary agreements about what truth looks like. When the underlying truth changes, the agreement is void.

Lesson: In some systems, correctness beats speed. If the cost of a stale read is a user who can't trust your product - kill stale entries the moment the source of truth changes.

The Boundary Rule

7) If two services share a DB → they’re one service wearing two badges

I watched a team celebrate their “microservices migration.” They had split their monolith into a User Service and an Account Service. Two repos. Two deploy pipelines. Two Slack channels for on-call. Everyone felt modern.

Both services read and wrote to the same users table in the same database.

What happened every schema change

• “We need to rename this column” → 2 PRs, 2 code reviews, 2 test suites

• Migration runs → sometimes only one service gets the update

• Rollback one → breaks the other

• Neither team wants to own the migration; both blame the other

This isn’t a microservices architecture. It’s a distributed monolith - all the operational complexity of microservices, none of the data isolation benefits.

A real service boundary is defined by data ownership, not repository count. If you can’t change one service’s schema without modifying the other - they are one service. Merge them, or properly separate the data.

Lesson: Shared data = shared fate. You can have separate codebases and still be a monolith at the data layer. True microservices have strict data ownership - not just separate repos.

Before you read further

I write content like this in much more depth on Skilled Coder

• 150+ Handpicked DSA problems

• Complete System Design Roadmap and 40+ HLD deep dives

• Complete LLD topics with Practical classical problems approaches

• Complete SQL + Data Engineering

• End to End Java Interview Preparation

• New Agentic AI foundation
  

  Single Library. Yours forever → theskilledcoder.com

The Root Cause Rule

8) If “just one more index” fixes it → fix the query shape, not the hardware.

A dashboard at a fintech startup kept timing out. The on-call rotation for that feature was becoming someone’s full-time job.

The first engineer added an index. Fast again. Everyone cheered.

A month later, another slow query - another index. Then another. Then another. Six months in, the transactions table had 14 indexes. Write operations noticeably slowed down because every insert was now maintaining 14 separate B-trees.

“We didn’t have a query problem. We had an index addiction.”

You finally sat down and profiled the queries seriously. Here’s what you found:

The actual problems, none of which needed an index

• Query joined on a non-selective UUID column (wrong join key)

• Filtered on status - only 3 possible values over 50M rows

• SELECT * fetching 40 columns when the UI used 4

• Wildcard LIKE '%keyword%' preventing index usage anyway

Now rewrote the queries. Dropped 9 of the 14 indexes. Performance improved. Write throughput improved.

How to actually fix it: Run EXPLAIN ANALYZE. Look for sequential scans on large tables. Fix the join key first. Then selectivity. Then column projection. Indexes are the last resort.

Lesson: Indexes are a bandaid. Query shape is the cure. Before adding another index, ask what the query is actually doing - and whether it should be doing it at all.

The Courage Rule

9) If every feature needs a flag → your release process is broken, not your code

You built a new onboarding flow. Clean design, better UX, everyone was proud of it. But to “reduce risk,” every step got wrapped in a feature flag.

The flag graveyard, after 6 weeks

• show_new_onboarding_form - on in prod, off in staging (no one knows why)

• enable_new_analytics_events - on, but the dashboard reading them is off

• new_validation_rules - off, because of a bug that was fixed 3 weeks ago

• enable_final_submission - nobody remembers what this was for

Toggling flags for each deploy was its own ceremony. QA had to test every flag combination. New engineers asked which flags were “safe” and got answers like: “...probably these ones?”

“We thought we were being safe. We were being afraid.”

The fix wasn’t better flag naming. It was fixing the conditions that made you afraid to ship without them. Once you had proper automated tests, proper staging parity, and gradual rollout infrastructure - you deleted most of the flags and never looked back.

Lesson: Feature flags are tools for confidence - not substitutes for it. If everything needs a flag, invest in your test coverage and your deploy pipeline. Ship with intent, not with hedges.

The best engineers aren’t the ones who never break things.
They’re the ones who break things once - and build something wiser in its place.

Every system scar is a lesson that no course could have taught you.

Thanks for reading :)

Share

Your app just logged this:

{
  "timestamp": "2024-01-15T14:32:47.892Z",
  "level": "INFO",
  "service": "checkout",
  "user_id": "u_8a7f3b",
  "action": "page_view",
  "page": "/products/shoes/running",
  "duration_ms": 47,
  "request_id": "req_9x8c7v6b"
}

Will anyone ever read this log entry? Probably not.

But you still wrote it. Indexed it. Replicated it. Paid to store it. Along with the other 47 million logs your system generated today.

This is the paradox of write-mostly systems: You must capture everything while knowing almost nothing will be read. The question isn’t IF you should store it, but HOW you store data optimized for writing, not reading.

Write-Mostly Systems in the Wild

• Application logs: 99.9% never viewed, critical for debugging that 0.1%

• Audit trails: Written for compliance, read only during audits

• Telemetry/metrics: Millions of data points, dashboards show aggregates

• Security events: Stored for forensics, queried during incidents

• IoT sensor data: Written constantly, analyzed in batches

• Backup/archive: Written once, hopefully never restored

This chapter is about designing systems that optimize for write throughput, minimize storage costs, and still provide fast access to the tiny fraction of data you’ll actually need - without knowing which fraction that is.

The Fundamental Principle: Stop Paying for Reads You’ll Never Do

Every traditional database feature that speeds up reads has a cost at write time:

If you’re writing millions of records that will rarely be queried, you’re paying the “read optimization tax” millions of times for essentially zero benefit.

Why We Ignore Traditional Database Design Here

Traditional databases optimize for balanced workloads. They assume you’ll query your data. For write-mostly systems, we deliberately break several “best practices”:

• No indexes at write time - We’ll build them later, if ever

• Eventual consistency is fine - We don’t need immediate reads

• Denormalization is expected - No joins means no foreign keys

• Data loss is acceptable (sometimes) - Sampling reduces volume

Principle 1: Append-Only Beats Random Writes

The fastest way to write data is to never update, only append. This isn’t just an optimization - it’s a fundamental shift in how we think about storage.

Why Append-Only is 10–100x Faster

Traditional databases do random writes: when you insert a row, it may go anywhere in the file based on indexes and ordering. This means:

• Disk seeks to find the right location

• Page splits when blocks are full

• Index rebalancing (B-tree rotations)

• Transaction logging for each operation

Append-only storage does sequential writes: new data always goes at the end. This means:

• No disk seeks (write head never moves backward)

• No page management (just append to the end)

• No index updates (we’re not indexing)

• Batched logging (bulk commit)

Principle 2: Store by Column, Not by Row

This is perhaps the most counterintuitive principle. Why would storing data in columns instead of rows make such a difference for write-mostly systems?

The Compression Advantage

In row-based storage, each row contains mixed data types: timestamps, strings, numbers, booleans. Mixed data compresses poorly.

In columnar storage, each column contains the same type of data. And here’s the key insight: log data is highly repetitive within columns.

Result: Columnar formats like Parquet achieve 10–50x compression on log data. What would be 500GB in JSON becomes 15–50GB in Parquet.

Why This Matters for Costs

At scale, storage dominates costs. Consider 1 billion logs per day:

Principle 3: Data Value Decays Over Time

Here’s a truth about write-mostly data: its value decreases exponentially with age. Today’s logs are critical for debugging a live issue. Last month’s logs are useful for trend analysis. Last year’s logs exist only for compliance.

Yet most systems store all data the same way, paying hot-storage prices for data that’s essentially frozen.

Tiered Storage: Match Cost to Value

The principle is simple: move data to cheaper storage as it ages. Accept slower query times for data that’s rarely queried.

The Hidden Cost: Retrieval Fees

Cold storage is cheap to store but expensive to retrieve. Before archiving:

Glacier Instant: $0.03/GB retrieval

Deep Archive: $0.02/GB retrieval + 12-hour wait

Design implication: If you expect frequent ad-hoc queries on historical data, cold storage may cost more than warm storage. Do the math for YOUR access patterns.

Principle 4: Index Later, Not Sooner

Traditional databases index at write time because they assume you’ll query immediately. For write-mostly systems, this is backwards.

The Lazy Indexing Philosophy

Don’t index what you won’t search. Since 99% of your data won’t be queried, indexing 99% is waste. Instead:

1. Write everything unindexed — Pure append, maximum speed

2. Index only high-value data — Errors, anomalies, specific transactions

3. Build indexes on-demand — When a query pattern emerges, create the index

Why this works: Modern columnar formats (Parquet) are surprisingly fast to scan. Querying 1 billion rows in a well-partitioned columnar dataset takes seconds, not hours. For rare queries, scanning beats the cost of maintaining indexes.

Principle 5: Accept That Some Data Can Be Lost

This is the most controversial principle. Traditional systems treat every record as sacred. But for many write-mostly use cases, statistical accuracy matters more than individual records.

When Sampling Makes Sense

A 10% sample of 100 million events (10 million rows) gives you statistically valid results for almost any aggregate query. You save 90% of storage and processing costs.

Related (Before you read further)

This article scratched the surface. The real interview goes 10 levels deeper.

How do you handle hot partitions?

What if the cache goes down during a spike?

How do you handle compliance requirements?

You don’t need 10 scattered resources anymore to prepare.

I’ve put HLD, LLD, DSA, Java, DBMS + real system-design scenario like this article plus real system-design questions broken down the way they’re actually asked in interviews.

👉 Check it out here

Putting It Together: The Write-Mostly Architecture

Case: When NOT to Use These Patterns

Don’t Apply Write-Mostly Patterns When:

• You need immediate reads: User activity feeds, real-time analytics where users see their own data

• Data requires updates: Status tracking, state machines, records that change over time

• Query patterns are unpredictable: BI tools where users slice data in arbitrary ways

• Consistency matters: Financial ledgers, inventory counts, anything where “eventually consistent” isn’t acceptable

What’s Next?

This problem can have variety of follow up questions

• How do you query data across tiers?

• What if you need to find a specific log entry?

• How do you handle compliance requirements?

• How do you decide retention periods?

We’ll cover similar real-world scenarios in upcoming articles or resource mentioned above.

Share

There’s a quiet shift that happens as you grow in your engineering career.
You stop asking “How do I make this work?” and start asking “What happens when this breaks?”

You begin to care less about clever one-liners of code and more about how a thousand moving parts talk to each other.
That’s when you stop thinking like “a developer” and start thinking like “an architect.”

Let’s look at the signs

1. You Think in Terms of Flows, Not Just Functions

Early in your career, you zoom into individual functions.
But an architect zooms out asking, “Where does this data go next? Who depends on it? What triggers it?”

A developer builds a “Send Invoice” function.
An architect asks, “When the invoice is sent, should we trigger a notification to finance? Should we update billing metrics? Should retries kick in if email fails?”

Architects see flows, not functions. They think in systems, not snippets.

2. You Optimize for Long-Term Change, Not Short-Term Speed

A developer says, “Let’s hack this and ship fast.”
An architect says, “This quick hack will haunt us every quarter.”

It’s not about slowing down it’s about building momentum that doesn’t collapse later.

You could hardcode a config today and save 30 minutes.
But you know six months later, marketing will need to change it weekly.
So you externalize it now. Because shipping fast is great shipping clean is better.

3. You Ask: “What Happens If It Fails?”

This is a big mindset jump.

When designing a feature, you’re not done when it works.
You’re done when you know what happens when it doesn’t.

You design a payment flow. An architect asks:

• What if payment API times out?

• What if webhook retries twice?

• What if user closes browser mid-transaction?

You plan graceful degradation not graceful ignorance.

Subscribe now

4. You Embrace Constraints, Not Just Features

Architects don’t see constraints as blockers. They see them as design boundaries.

You’re building a search API.
You don’t say, “We’ll return all results.”
You say, “We have a 200ms SLA and a 100MB memory cap how do we stay inside that box?”

Real-world architecture is art within limits, not freedom without trade-offs.

5. You Design for Team Boundaries, Not Just Code Boundaries

Good code is modular.
Great architecture is organizationally modular.

If 3 teams touch the same service to deploy, you’ve built a bottleneck, not a service.
Architects design for ownership so teams can ship without stepping on each other’s toes.

Clean boundaries create independent velocity.

6. You Think in Contracts & Interfaces

In an architect’s world, APIs are contracts, not handshakes.
You don’t just say “call this endpoint” you define what’s guaranteed, what’s optional, and what’s versioned.

When you change a field name in a schema, you don’t “just fix it” you create a new version.
Because backward compatibility isn’t optional it’s respect for other systems.

7. You Separate What Changes from What Doesn’t

Architects spot volatility early.
They ask: “What’s likely to change?” and isolate it.

If your pricing logic keeps changing, don’t bury it deep inside order processing.
Put it in a strategy layer or a config-driven rule so the rest of the system stays stable.

It’s not about predicting the future. It’s about shielding yourself from it.

8. You Question Dependencies Before Adding Them

Every new library is a future meeting with your past self.
Architects ask: “Will this still be maintained in 2 years? Does it fit our stack? Can we remove it easily?”

You might pick a fancy caching library today but if it blocks Java 22 migration later, was it worth it?

Architects treat dependencies like guests, not roommates.

9. You Build for Observability Upfront

If a system fails and nobody can explain why, that’s not a bug that’s bad design.

You don’t add logs after an outage. You design with logs, metrics, and traces from day one.
You know every critical path should leave breadcrumbs.

Because good systems talk back when they’re in trouble.

10. You Default to Asynchronous Where It Makes Sense

Architects understand the power of decoupling.

Instead of making a user wait for an email to send, you enqueue a job.
Instead of chaining 5 API calls, you publish an event and let subscribers act.

You build for scale, not sequence.

11. You Care About Data Contracts

Data isn’t “just stored” it’s a living interface between services.

You don’t just alter a table column. You think:

• What breaks downstream ETL?

• Are old events still valid?

• Should I add a new column instead of mutating old one?

Architects know data is forever, so they tread carefully.

12. You Minimize Coordination, Not Just Code

Architecture isn’t about reducing lines of code it’s about reducing lines of communication.

If two teams need a sync meeting every time they deploy, your design failed.
Independent deployability > shared spreadsheets.

You aim for loosely coupled teams, not just loosely coupled code.

13. You Think in Layers

Architects layer systems like cities:

• Infrastructure → Roads

• Business logic → Buildings

• Interfaces → Doors & Windows

• Observability → CCTV

Each layer has purpose, boundaries, and rules. You don’t mix roads with roofs.

Your logging config doesn’t live in your controller logic it lives where it belongs: infra.

14. You Know Simplicity Is the Hardest Part

Anyone can add complexity.
Architects earn their stripes by removing it.

You resist the urge to use 4 queues, 3 caches, and a quantum database.
You ask, “Can this be a cron job instead?”

Because elegance isn’t about how much you can build it’s about how little you need to.

Becoming an architect isn’t a title. It’s a mindset.

It’s when you stop thinking about features and start thinking about systems.
It’s when you stop coding for today’s demo and start designing for tomorrow’s change.

Related

If you enjoy digging into how things actually work, I’ve been working on something larger.

I wrote an ebook that walks through 35 real engineering problems and system designs, focusing on how decisions break down at scale, what trade-offs actually matter, and the kinds of follow-up questions engineers run into in interviews and real systems.

It’s less about memorizing solutions, and more about learning the principles behind solving problems at scale - so you can reason through new situations, not just familiar ones.

👉 Check it out here

Share

Taylor Swift posts a single tweet: “New album at midnight”

That’s one write. One row inserted.

Within 30 seconds, 90 million followers need to see it in their timeline. That’s 90 million reads. For every single tweet.

When reads outnumber writes 1000:1, you can’t afford to compute anything at read time. The answer must already be waiting.

This chapter is about systems where you intentionally make writes slow and expensive so that reads become blazingly fast.

The economics are simple: if you read 1000x more than you write, spend 1000x more effort on the write.

First Attempts (and Why They Don’t Scale)

Attempt 1: Query at Read Time

-- Get user’s feed: JOIN everything at read time
SELECT posts.*
FROM posts
JOIN follows ON posts.author_id = follows.following_id
WHERE follows.follower_id = :user_id
ORDER BY posts.created_at DESC
LIMIT 50;

Why it seems right:

• Normalized data - no duplication

• Always fresh - up to the second

• Simple to understand

Why it fails at scale:

• Massive JOINs: User follows 500 people × 100 posts each = scanning 50,000 rows

• Repeated work: 10 users follow the same celebrity? Same computation 10 times

• Read latency: 200ms+ per request when database is under load

• Database meltdown: Can’t scale reads without replication lag issues

Attempt 2: Add a Cache

def get_feed(user_id):
    cached = redis.get(f”feed:{user_id}”)
    if cached:
        return cached
    
    # Cache miss - do the expensive query
    feed = expensive_feed_query(user_id)
    redis.setex(f”feed:{user_id}”, 300, feed)  # 5 min TTL
    return feed

Why it fails:

• Cache stampede: When cache expires, 1000 users hit DB simultaneously

• Stale data: New posts don’t appear for 5 minutes

• Cold start: New users or cache eviction = slow first load

• The underlying problem persists: Still doing the expensive query, just less often

The Cache is a Band-aid : Caching a slow query doesn’t fix the query. It just hides the problem… until the cache misses. The real solution is to never run the slow query at all.

The Patterns That Actually Work

Pattern 1: Fan-Out on Write

Instead of computing the feed at read time, push posts to followers at write time.

Trade-offs:

Pattern 2: Hybrid Fan-Out (The Twitter Solution)

For celebrities with millions of followers, pure fan-out is too expensive. Solution: fan-out for regular users, pull for celebrities.

def post_tweet(author_id, content):
    tweet = db.insert(author_id=author_id, content=content)
    follower_count = get_follower_count(author_id)
    
    if follower_count < 10000:  # Regular user
        # Fan out to all followers
        for follower_id in get_followers(author_id):
            redis.lpush(f”feed:{follower_id}”, tweet.id)
    else:  # Celebrity - don’t fan out
        # Just mark the tweet as from a celebrity
        redis.sadd(”celebrity_tweets”, tweet.id)

def get_feed(user_id):
    # 1. Get pre-built feed (from regular users)
    feed = redis.lrange(f”feed:{user_id}”, 0, 50)
    
    # 2. Merge in celebrity tweets (computed at read time)
    celebrity_ids = get_followed_celebrities(user_id)
    for celeb_id in celebrity_ids:
        recent_tweets = get_recent_tweets(celeb_id)
        feed = merge_and_sort(feed, recent_tweets)
    
    return feed[:50]

Why this works: 99% of users have < 10K followers. Fan-out handles them. The 1% of celebrities require a small read-time merge, which is acceptable.

Pattern 3: Denormalization

Store data redundantly so reads don’t require JOINs.

// Normalized: Requires JOIN to get author name
{
  “post_id”: 123,
  “author_id”: 456,
  “content”: “Hello world”
}

// Denormalized: Everything needed is in one place
{
  “post_id”: 123,
  “author_id”: 456,
  “author_name”: “Taylor Swift”,
  “author_avatar”: “https://...”,
  “author_verified”: true,
  “content”: “Hello world”,
  “like_count”: 47000000,
  “reply_count”: 890000
}

The cost: When Taylor changes her avatar, you update millions of posts.

The benefit: Every read is a single key lookup. No JOINs. Ever.

When to Denormalize

• Data is read 100x more than written

• The denormalized fields rarely change (names, avatars)

• Read latency is critical

• You can tolerate eventual consistency (old avatar for a few minutes)

Pattern 4: Materialized Views

Pre-compute and store query results. Update them when underlying data changes.

Use cases:

• Dashboard aggregations (daily sales, weekly signups)

• Leaderboards (top sellers, trending content)

• Recommendation scores (pre-computed similarity)

-- Create a materialized view of product stats
CREATE MATERIALIZED VIEW product_stats AS
SELECT 
    product_id,
    COUNT(*) AS review_count,
    AVG(rating) AS avg_rating,
    SUM(helpful_votes) AS total_helpful
FROM reviews
GROUP BY product_id;

-- Read: instant (no aggregation)
SELECT * FROM product_stats WHERE product_id = 123;

-- Refresh periodically or on trigger
REFRESH MATERIALIZED VIEW product_stats;

Pattern 5: CQRS (Command Query Responsibility Segregation)

Separate the write model from the read model entirely. Two different databases, optimized for their specific purpose. (image in header)

The key insight: The read model is a projection. It can be deleted and rebuilt from the write model anytime. This frees you to optimize it purely for reads.

Architecture: Putting It Together

Data Flow: Write Path

1. User creates a post → write to primary database

2. Publish event to message queue

3. Fan-out worker: push to followers’ feed caches

4. Denormalization worker: update cached objects with new data

Data Flow: Read Path

1. User opens app → hit CDN for static content

2. API request → check Redis cache first

3. Cache hit (99% of time) → return immediately

4. Cache miss → read from replica, populate cache, return

Interview Pivot (How to Explain It)

If asked:

“How do you design a system where reads are cheap but writes are expensive?”

“The core principle is to move computation from read time to write time. If data is read 1000x for every write, I should do the expensive work once during the write, not 1000 times during reads.

For a feed system, I’d use fan-out on write. When a user posts, I push the post ID to every follower’s pre-built feed list in Redis. Reads become a simple list retrieval - O(1) instead of complex JOINs.

For celebrities with millions of followers, I’d use hybrid fan-out: push for regular users, pull for celebrities. We merge celebrity tweets at read time, which is acceptable for the small percentage of celebrity follows.

I’d also denormalize aggressively. Store author name and avatar with every post so reads don’t require JOINs. Yes, updating a user’s avatar means updating millions of posts, but that’s a background job - it doesn’t affect read latency.

For complex aggregations, I’d use materialized views or CQRS - separate read and write models entirely. The read model is denormalized and optimized purely for query patterns. It’s eventually consistent with the write model, but for most use cases that’s fine.”

What’s Next?

Every fast read is backed by expensive writes. There’s no magic - just a choice of when to pay the cost.

For read-heavy systems, pay at write time. Pre-compute feeds, denormalize data, maintain materialized views. Your writes become complex, but your reads become trivial. And at 1000:1 read/write ratio, that’s exactly the trade you want.

Read-heavy systems lead to deeper questions:

• How do you handle a user unfollowing someone?

• What if the write model and read model get out of sync?

• How do you handle a new user with no pre-built feed?

• How much storage does fan-out actually use?

Related

This article scratched the surface. The real interview goes 10 levels deeper.

• How do you handle hot partitions?

• What if the cache goes down during a spike?

• How do you avoid counting the same view twice?

I’ve written an ebook that prepares you for all of it. (In detail with follow ups)

35 real problems. The patterns that solve them. The follow-ups you’ll actually face. The principles behind solving problems at scale, not just the final answers.

👉 Check it out here

Share

Every engineering team reaches a moment where things start to creak. Requests spike unpredictably. Background jobs lag behind. One slow dependency begins dragging everything else down. That's usually the moment someone says: "Should we introduce Kafka?"

Sometimes that suggestion saves the system. Other times, it quietly makes everything worse. To know the difference, you need to understand the fundamental principle behind message queues.

Before Kafka, before brokers and partitions, there is a simple truth: Different parts of a system run at different speeds. The moment you force them to move at the same pace, you create fragility.

Think about it this way: Your API server can accept 50,000 requests per second because it's just receiving data. But your database can only write 5,000 records per second because it needs to persist data to disk, maintain indexes, and ensure consistency. When you directly connect the fast component to the slow component, the slow one becomes a bottleneck that drags everything down.

Queues exist to decouple time: Producers move fast, consumers move at their own pace, and the system survives bursts, failures, and change. Kafka is one powerful way to achieve this decoupling - but power only helps when you're solving the right problem.

When You SHOULD Use Kafka

1. When Producer Speed ≠ Consumer Speed

This is the most fundamental use case for any message queue, and it's where Kafka truly shines.

The Real-World Scenario: Imagine you’re running an e-commerce platform. Your API servers are designed to handle high concurrency - they accept HTTP requests, validate input, and return responses quickly. On a normal day, you might receive 50,000 requests per second during peak hours.

But here’s the problem: Your downstream systems aren’t that fast. Your database needs time to write records (disk I/O, index updates, replication). Your payment processor has rate limits. Your inventory service checks stock levels against a database. Each of these can only handle perhaps 5,000-10,000 operations per second.

Kafka uses an append-only log structure. Writing to Kafka is essentially just appending bytes to a file, which is incredibly fast. Your API can acknowledge 50,000 requests per second because it's not waiting for slow downstream processing. The consumers then read from Kafka at whatever pace they can sustain.

Any time one side of your system is faster than the other, a queue acts as a buffer that absorbs the difference.

2. When Traffic Spikes Are Unpredictable

Some systems experience steady, predictable traffic. Others face dramatic spikes that can appear without warning. Kafka excels at the latter.

The Flash Sale Scenario: Your e-commerce platform announces a flash sale at noon. At 11:59, you're handling 1,000 requests per second. At 12:00:01, it jumps to 100,000 requests per second as eager shoppers rush to grab deals. Three minutes later, it's back to 5,000 requests per second.

More Examples like Flash Sales, Marketing Campaigns, OTP Generation, Push Notifications, Log Ingestion etc

Why This Is Hard: If you provision infrastructure for peak load (100K/sec), you're wasting 99% of your resources during normal operation. But if you provision for normal load, you'll crash during the spike.

How Kafka Helps: During the spike, Kafka absorbs all 100,000 requests per second into its log. Yes, the queue depth grows temporarily. But your consumers keep processing at their steady 10K/sec rate. Within a few minutes after the spike ends, the queue drains back to normal.

Meanwhile, you can configure auto-scaling to add more consumers when queue depth increases, but this happens gradually over minutes - not the split-second response you’d need without a queue.

The Math: A 3-minute spike at 100K/sec produces 18 million messages. At 10K/sec consumption, it takes 30 minutes to clear. Your system survives because Kafka holds that buffer, not because you magically scaled 10x in milliseconds.

3. When Work Is Asynchronous by Nature

Some operations don't need to complete before responding to the user. Making users wait for these operations is both bad UX and wasteful system design.

The Email Scenario: A user signs up for your service. You need to send a welcome email. Should the user stare at a loading spinner while your server connects to the SMTP server, composes the email, and waits for delivery confirmation?

Similarly for Push Notifications, Image/Video Processing, PDF Generation, Analytics Events

Why Sync is Wrong Here: Sending an email might take 1-3 seconds (DNS lookup, SMTP handshake, server response). That’s 1-3 seconds the user waits unnecessarily. Worse, if the email server is slow or down, the user’s signup fails - even though email delivery isn’t critical to completing signup.

The Async Pattern: Your API pushes a “send welcome email” event to Kafka and immediately returns success to the user. A background worker picks up that event and handles the actual email sending. The user gets instant feedback, and the email arrives moments later.

If the user doesn't need the result immediately, don't make them wait. Push the work to a queue and respond instantly.

4. When Reliability Matters More Than Speed

Some events absolutely cannot be lost. A payment confirmation, an order placement, a financial transaction - these must be processed eventually, even if systems fail temporarily.

The Payment Scenario: A customer completes a payment. You need to: update their order status, send a confirmation email, notify the warehouse, update analytics, and trigger loyalty points. If any of these downstream services are down at that moment, what happens to the payment event?

More cases Order Confirmations, Financial Transactions, Audit Logs, Compliance Events

Kafka's Durability: When you produce a message to Kafka with proper acknowledgment settings, that message is written to disk and replicated across multiple brokers. Even if the consumer is down for hours, the message waits patiently. When the consumer comes back, it picks up right where it left off.

If losing an event is unacceptable, a durable queue like Kafka provides the safety net. The event is guaranteed to be processed eventually, even through failures.

5. When Multiple Consumers Need the Same Event

This is where Kafka’s pub/sub model truly shines - the ability to broadcast one event to many independent consumers without the producer knowing or caring about them.

The Order Scenario: When a customer places an order, multiple systems need to react: Inventory needs to reserve stock, Notifications needs to send confirmation, Analytics needs to track the sale, Fraud Detection needs to evaluate risk, Warehouse needs to prepare shipping, and Loyalty needs to add points.

Without Kafka: The Order Service would need to know about every downstream service. Adding Loyalty Points? Change Order Service code. Adding Fraud Detection? Change Order Service code again. Each addition increases coupling and deployment risk.

With Kafka: Order Service publishes one event to the “orders” topic and walks away. Each downstream service subscribes to that topic independently. Adding a new consumer? Zero changes to Order Service. Each team owns their own consumer.

One event, many independent consumers, zero producer changes. Teams can evolve independently, deploy independently, and scale independently.

Other Scenarios Where Kafka Fits

• When consumers evolve independently: Different teams work on different consumers with different release cycles.

• When ordering matters: Kafka guarantees order within a partition - critical for state machines and sequential processing.

• When retry logic would be ugly: Instead of implementing retry everywhere, let Kafka handle redelivery.

• When you need horizontal scaling: Add more consumers to a consumer group to handle more load.

• When you want replay capability: Kafka retains messages, letting you reprocess historical events for debugging or recomputation.

If direct service-to-service calls feel fragile, slow, or scary - you likely need a queue.

Related (Before reading further)

If you enjoy digging into how things actually work, I’ve been working on something larger.

I wrote an ebook that walks through 35 real engineering problems and system designs, focusing on how decisions break down at scale, what trade-offs actually matter, and the kinds of follow-up questions engineers run into in interviews and real systems.

It’s less about memorizing solutions, and more about learning the principles behind solving problems at scale - so you can reason through new situations, not just familiar ones.

Check System Design Book Here

When You Should NOT Use Kafka

This is the part people skip and later regret. Kafka adds complexity. Using it when you don't need it creates operational burden without corresponding benefit.

1. When You Need Immediate Request-Response

Kafka is fundamentally asynchronous. It’s built for “fire and forget” patterns, not for “call and wait for response” patterns.

The Login Scenario: A user enters their username and password and clicks "Login". They need an immediate answer: Are credentials valid? What's their session token? This must happen in milliseconds.

Similarly User Profile Fetch, Balance Check, Search Queries, Real-time Validation

Why Kafka Fails Here: To use Kafka for request-response, you’d need to: publish a request to Topic A, have a consumer process it, publish a response to Topic B, and have the original requester consume from Topic B. This adds 100-500ms of latency, requires correlation IDs to match responses, and creates complex error handling.

Use Instead: Direct HTTP/gRPC calls or database reads. If the user is staring at the screen waiting, Kafka is the wrong tool.

2. When Traffic Is Low and Predictable

Kafka’s power comes from handling scale and unpredictability. At low, steady traffic, it’s just overhead.

The Small App Scenario: Your internal tool handles 50-200 requests per second. Traffic is steady during business hours and drops to zero at night. You have a few background jobs that run every few minutes.

Internal Tools, Admin Dashboards, Small B2B Apps, MVPs etc

The Kafka Tax: Running Kafka means: managing a broker cluster (minimum 3 for production), configuring partitions and replication, monitoring consumer lag, handling broker failures, managing disk space and retention, and training your team on Kafka operations.

For 50 requests per second, this operational burden far exceeds any benefit. A simple database table with a worker polling every few seconds would handle your needs with zero additional infrastructure.

Use Instead: Cron jobs, simple database-backed job queues, or lightweight queues like Redis. Save Kafka for when you actually have scale problems.

3. When You Only Have One Producer and One Consumer

Kafka’s real value emerges with decoupling and fan-out. With a single producer and single consumer, you’re adding complexity without gaining the benefits.

The Middleman Problem: Service A needs to send data to Service B. You add Kafka between them. Now you have: two services to debug instead of direct call tracing, additional failure mode (what if Kafka is down?), consumer lag to monitor, and no real benefit since there’s no fan-out or speed mismatch.

Use Instead: Direct async HTTP calls, or lightweight queues like SQS or RabbitMQ if you need retry/durability. Kafka's power is wasted in 1:1 scenarios.

4. When Durability and Replay Don’t Matter

Kafka persists everything to disk. That’s its strength - but also overhead when you don’t need it.

Ephemeral Events: You’re broadcasting cache invalidation signals. You’re sending “best effort” metrics that are fine to lose occasionally. You’re coordinating temporary state between services. In all these cases, if an event is lost, it’s not a disaster — the next event will correct things.

The Overhead: Kafka writes every message to disk, replicates it across brokers, and retains it based on your configuration. For fire-and-forget events, this is wasted I/O and storage.

Use Instead: In-memory queues, Redis pub/sub or streams, or simple fire-and-forget HTTP calls. Match your tool to your durability requirements.

5. When Tasks Are Simple and Ordering Doesn’t Matter

Kafka guarantees ordering within partitions, but this guarantee comes with complexity. If you don’t need ordering, simpler tools work better.

Independent Tasks: You need to resize 10,000 uploaded images. Each image is independent - the order doesn’t matter. You need to send 50,000 emails - each is independent. For these “embarrassingly parallel” workloads, Kafka’s partitioning model adds unnecessary complexity.

Use Instead: Purpose-built task queues like SQS, Sidekiq, Celery, or BullMQ. These are simpler to operate and designed specifically for independent job processing.

More Warning Signs

• Team can’t operate Kafka: Kafka requires operational expertise. Without it, you’ll face outages.

• “FAANG uses it” reasoning: Cargo-culting doesn’t solve problems. FAANG operates at scales you probably don’t.

• Large message payloads: Kafka is optimized for events (KB), not blobs (MB). Use object storage for large files.

• Ultra-low latency requirements: If you need sub-millisecond latency, Kafka’s inherent latency (10-100ms) may be too high.

Kafka solves scaling pain, not design confusion. If your system is already messy, Kafka will just make the mess distributed.

Share

One quiet Friday you ship your new app, close the laptop, and head out for coffee. Suddenly a celebrity tweets your link. In ten minutes 10,000 people smash the Sign Up button - and your single‑box server starts doing its best impression of a pressure cooker.

At first, it sounds like a capacity question.

In reality, it’s a systems thinking question.

Let’s start with the approaches that seem correct - and why they fail.

First Attempts (and Why They Break)

Attempt 1: Just Add More Servers

The most common instinct. Traffic increases → add more app servers → problem solved.

Why it feels right:

• Horizontal scaling is industry standard

• Cloud makes it easy

• Works at low to medium traffic

Why it fails:

• Shared bottlenecks don’t scale automatically

• Database, cache, auth service, third-party APIs all become choke points

• One slow downstream service can stall thousands of requests

You scaled the frontend, not the system.

Lesson :Concurrency pressure usually breaks dependencies, not CPU.

Attempt 2: Increase Thread Pool Size

Another classic fix. “Let’s increase the thread pool from 200 to 2,000.”

Why it feels right:

• More threads = more users handled

• Easy config change

• Looks like progress

Why it fails:

• Threads consume memory and context-switching time

• CPU spends more time scheduling than doing work

• Latency gets worse, not better

• JVM / runtime starts thrashing

Concurrency is not free just because threads exist.

Lesson : More threads don’t mean more throughput.

Attempt 3: Cache Everything

Okay, let’s cache aggressively. Cache DB responses, API calls, user sessions.

Why it feels right:

• Reduces load

• Improves latency

• Caching works - until it doesn’t

Why it fails:

• Cache stampedes under concurrent misses

• Hot keys get hammered

• Cache invalidation logic becomes fragile

• Memory pressure increases fast

Caching helps load, but it doesn’t control concurrency.

Lesson : Cache reduces work, it doesn’t manage access.

Attempt 4: Let Requests Queue Up

Another tempting idea: “If users come faster than we can handle, let them wait.”

Why it feels right:

• No requests dropped

• Everything is eventually processed

Why it fails:

• Queues grow without bound

• Latency explodes

• Timeouts cascade across services

• Users retry → doubling traffic

Unbounded queues turn traffic spikes into outages.

The Real Problem

10,000 concurrent users is not about users.

It’s about how many things your system can do at the same time without stepping on itself.

The real questions are:

• Which operations are blocking?

• Which resources are shared?

• Which paths must be fast, and which can be delayed?

Once you answer those, the solution becomes clearer.

Lets see step by step how to build such system

Related [Before reading further]

This article scratched the surface. The real interview goes 10 levels deeper.

• How do you handle hot partitions?

• What if the cache goes down during a spike?

• How do you avoid counting the same view twice?

I’ve written an ebook that prepares you for all of it.

35 real problems. The patterns that solve them. The follow-ups you’ll actually face. The principles behind solving problems at scale, not just the final answers.

👉 Check it out here

The Correct Architecture

1. Make the Request Path Thin

The request thread should do as little as possible.

• Validate input

• Authorize

• Enqueue work

• Return response

Anything slow or non-critical moves out of band.

Concurrency survives when requests finish fast.

2. Control Concurrency Explicitly

Instead of letting everything run freely:

• Limit concurrent DB calls

• Limit outbound API calls

• Apply bulkheads per dependency

When limits are hit:

• Fail fast

• Return degraded responses

• Don’t let pressure spread

This protects the system from itself.

3. Separate Immediate Work from Deferred Work

Not everything needs to happen now.

• Emails

• Notifications

• Analytics

• Logging

• Recommendations

Push these to queues and process asynchronously.

Users care about response time, not background perfection.

4. Scale State, Not Just Compute

Stateless app servers scale easily. State doesn’t.

• Partition data

• Reduce shared locks

• Avoid global coordination

• Prefer per-user or per-shard isolation

Concurrency collapses when everyone waits on the same thing.

5. Apply Backpressure Early

When the system is under stress:

• Reject requests early

• Return 429s

• Degrade features

• Shed load intentionally

A fast rejection is better than a slow failure.

Why This Works

This approach works because it accepts a hard truth:

• Requests finish quickly

• Slow work happens elsewhere

• Failures stay contained

• Traffic spikes don’t turn into outages

10,000 concurrent users stop being scary. They become just another number.

Interview Pivot

If asked:

“How would you handle 10,000 concurrent users?”

“I’d keep the request path thin, control concurrency to shared dependencies with bulkheads, move slow work async, apply backpressure early, and scale state carefully.

That way concurrency increases without collapsing latency.”

That answer shows you understand system behavior, not just scaling slogans.

What’s Next?

Real systems bring follow-ups:

• What happens when one dependency slows down?

• How do you prevent retry storms?

• How do you degrade gracefully?

• How do you observe concurrency bottlenecks?

We’ll dive deeper into these kinds of follow-ups in upcoming articles (or in ebook)- subscribe if you want more.

Thanks for reading. See you in next post.

Share

Every social platform has it. A Trending section.

The posts everyone seems to be talking about right now - not yesterday, not last week, but this minute.

At first glance, it sounds easy. Just count likes and sort, right?

That illusion lasts until traffic grows, events start arriving out of order, and one viral post suddenly floods your system. What felt like a simple ranking problem quickly turns into a real-time data challenge.

In This Chapter

• The solutions that look correct but break at scale

• Why “trending” is not a batch problem

• How real systems compute it continuously using streaming data

We will solve it with first-principles thinking.

First Attempts (and Why They Break)

Attempt 1: Recompute Everything Every Minute

The most obvious idea is this:

1. Every minute, scan all recent likes, comments, shares, and views

2. Group them by post

3. Sort by count

4. Pick the top 10

Simple SQL or Spark job. Done.

Why it feels right:

• Easy to reason about

• Works fine when you have 10k posts

• “Trending every minute” sounds like a batch problem

Why it fails:

• You’re reprocessing the same data again and again

• At scale, each minute means scanning millions of events

• Latency creeps up → your “every minute” job takes longer than a minute

• Spikes (viral posts) make jobs miss their SLA

What looked like a clean batch job quietly turns into a runaway compute bill.

Attempt 2: Maintain Counters in the Database

Okay, let’s be smarter.

Whenever a user likes or comments on a post: Increment a counter in the posts table. Use that counter to decide what’s trending.

Why it feels right:

• Constant-time updates

• Data is always “fresh”

• Easy queries like ORDER BY score DESC

Why it fails:

• High write contention on hot posts

• Viral posts become bottlenecks

• Locks, retries, and replication lag start showing up

• Your primary DB is now handling analytics traffic

Suddenly, your transactional database is doing real-time ranking. That never ends well.

Attempt 3: Cache Trending Posts in Redis

Fine. Move fast stuff to Redis.

• Increment counters in Redis

• Every minute, read top keys

• Redis is fast, right?

Why it feels right:

• In-memory speed

• No DB locks

• Widely used pattern

What’s the problem:

• Redis becomes a single hot spot

• You lose ordering guarantees under heavy concurrency

• Crash or restart = counters gone (unless heavily persisted)

• Cross-region or multi-shard ranking gets messy

Redis helps performance, but it doesn’t solve time-windowed aggregation cleanly.

Attempt 4: Trigger Jobs with Cron

Another classic move:

• Run a cron job every minute

• Compute trending posts for the last 60 seconds

• Publish the result

Why it feels right:

• Simple mental model

• Clear time boundaries

• Easy to debug

What’s the issue:

• All computation spikes at the same second

• Late events get dropped or miscounted

• Real-time systems don’t respect clock boundaries

• Users see inconsistent “trending” lists

Trending is a flowing signal, not a punctual snapshot.

Attempt 5: “Just Use Machine Learning”

Someone always suggests this. “Why not train a model to predict trending posts?”

Why it feels right:

• Sounds sophisticated

• Great for long-term ranking

Why it fails (for this problem):

• ML still needs clean, real-time features

• You still haven’t solved streaming aggregation

• Latency requirements make inference tricky

• Overkill for minute-level trends

ML improves ranking. It doesn’t replace real-time computation.

The Pattern Emerging

All these approaches fail for the same reason:

They treat trending as a static calculation.
In reality, trending is a moving window over a live stream of events.

To solve it properly, we need to stop thinking in rows and start thinking in flows.

The Right Way to Think About Trending

Once you step back, the mistake in all the earlier attempts becomes obvious.

That immediately tells us two things:

1. We must process events as they happen

2. We must reason in sliding time windows, not batches or cron jobs

This is a streaming problem, not a database one.

The Mental Model Shift

Instead of asking:

“Every minute, what are the top posts?”

We ask:

“As events flow in, how does each post’s momentum change?”

Once you adopt this mindset, the architecture almost designs itself.

The Architecture: Step by Step

Step 1: Treat Every Interaction as an Event

Likes, comments, shares, views - they are all signals. Each interaction becomes a small event:

{
  "post_id": "abc123",
  "event_type": "like",
  "timestamp": 1702999200
}

Nothing is aggregated yet. Nothing is ranked yet. We simply emit events and move on.

This keeps the write path extremely fast and lets the system absorb sudden spikes when something goes viral.

Step 2: Stream, Don’t Store First

These events flow into a stream - Kafka, Pulsar, Kinesis, Pub/Sub.

Why a stream? Because streams:

• Preserve order within partitions

• Handle burst traffic naturally

• Allow multiple consumers to compute different views

At this point, the system isn’t “computing trending” yet. It’s just recording reality as it unfolds.

Step 3: Compute Trends Using Sliding Windows

Now comes the core idea.

Instead of recomputing everything every minute, we maintain rolling windows:

• Last 1 minute

• Last 5 minutes

• Last 15 minutes

As events flow in, stream processors continuously update counters per post per window.

This is not batch. This is incremental math.

When a new like arrives:

1. Add +1 to that post’s current window

2. Expire events that fall out of the window

3. Update the score instantly

No rescans. No cron jobs. No waiting.

Trending becomes a continuously updated result.

Step 4: Separate Scoring from Ranking

Raw counts alone don’t tell the full story.

A post with 10 likes in 1 minute might be more interesting than one with 1,000 likes over a day.

So instead of just counting, we compute a score:

• Weight recent events higher than older ones

• Give comments more weight than likes

• Normalize by post age

This scoring logic lives inside the stream processor.

The output is simple:

{
  "post_id": "abc123",
  "trend_score": 847.5,
  "window": "5min"
}

The heavy thinking happens once, in the stream - not on every read.

Step 5: Maintain a Live Top-K Structure

From these scores, we maintain a Top-K list (say top 100 posts per region or category).

This structure updates incrementally:

• When a score changes, we adjust its position

• No full sorting required

• No global locks

The result is always ready.

When the UI asks for “trending posts”, it’s a simple read - not a computation.

Step 6: Serve Reads Without Touching the Stream

The final trending list is pushed to a fast store:

• Redis

• DynamoDB

• In-memory cache

Reads are cheap. Writes are controlled. The stream keeps flowing independently.

If the UI goes down, streaming continues. If the stream lags, the UI still serves the last known good state.

Failures don’t cascade.

Why This Works (At Scale)

This architecture works because nothing waits.

• Events flow in once

• Computation happens incrementally

• Ranking updates continuously

• Reads never trigger heavy work

You’re no longer fighting time or traffic spikes.

Trending stops being a periodic job and becomes a living signal.

Interview Pivot (How to Explain It)

If someone asks:

“How would you compute trending posts every minute?”

“I wouldn’t recompute every minute. I’d treat user interactions as a stream, aggregate them in sliding windows, compute a rolling trend score per post, and maintain a live top-K list.

The UI simply reads the latest result, while the stream updates continuously in the background.”

That answer shows you understand data flow, not just data storage.

What’s Next?

Trending isn’t something you calculate on a schedule. It’s something you observe as it emerges.

This design opens the door to deeper questions:

• How do you handle late or out-of-order events?

• How do you avoid one viral post dominating forever?

• How do you scale this across regions?

• How do you debug wrong trending results?

The key insight: Trending isn’t about counting - it’s about capturing momentum. The systems that get this right don’t fight the stream of events. They ride it.

Think in flows, not snapshots. That’s how real-time systems scale.

Related

This article scratched the surface. The real interview goes 10 levels deeper.

• How do you handle hot partitions?

• What if the cache goes down during a spike?

• How do you avoid counting the same view twice?

I've written an ebook that prepares you for all of it.

35 real problems. The patterns that solve them. The follow-ups you'll actually face. The principles behind solving problems at scale, not just the final answers.

👉 Grab your copy here

Thanks for reading. See you in next post.

Every modern system - whether it’s a fintech app processing payments, an ad-tech platform tracking impressions, or a multiplayer game updating player actions - relies on one silent backbone:

Logs

They’re the breadcrumbs of truth.

• A debit failure in banking

• A player teleporting in a game

• A “Buy Now” button click in e-commerce

• A leaderboard refresh in a fantasy sports app

Individually, they look tiny.

Collectively, they become millions per second.

The issue is

Logging feels trivial when traffic is small.
At scale, logging becomes one of the most expensive operations in your system.

Because logging is not “just write a line.”

It is I/O + storage + latency + durability + retention + indexing + retrieval + analytics.

And once your traffic crosses a certain threshold, the naïve approach falls apart.

Let’s break down why logging at massive scale is hard, and how to do it right.

First Attempt: Log Directly Inside App Code

Most beginners start here:

log.info(”User “ + userId + “ clicked Buy Now”);

Why it feels right:

• Easy to write

• Works in dev

• Shows logs instantly

Why it fails at scale:

• Logging is blocking IO

• Every write competes with CPU cycles meant for business logic

• If logs go to disk → disk bandwidth becomes bottleneck

• If logs go to external system → network latency kills throughput

• If logs are synchronous → one slow write pauses the entire request

At 1M events/sec, this model collapses.

Even a 1ms logging cost means:

1,000,000 events × 1ms = 1,000,000ms = 1000 seconds latency → impossible.

Direct synchronous logging couples application performance with logging performance and that’s dangerous.

Attempt 2: Batch Logs and Write Periodically

Okay, so developers try batching:

buffer.append(event)
if buffer.size >= 1000:
    writeToDisk(buffer)
    buffer.clear()

Why it feels right:

• Fewer writes = fewer syscalls

• Better throughput

• Common in logging libraries like Logback, Bunyan, Serilog etc.

Why it still breaks:

• If your process crashes → buffered logs vanish

• If bursts exceed buffer capacity → logs drop

• Writing periodically still requires locking, causing contention across threads

• If storage slows down, buffers back up → memory explodes

At 1M/sec, even buffering isn’t enough.

Batching helps, but doesn’t decouple logging from request handling.

 ┌──────────────────────┐
 │   Application Code   │
 └───────────┬──────────┘
             │
             │
             ▼
       (In-memory buffer)
        ┌───────────────┐
        │   Log Buffer  │
        │  (Batch size) │
        └───────┬───────┘
                │
       (flush every X events/time)
                ▼
       ┌─────────────────┐
       │  Disk/DB/HTTP   │
       └─────────────────┘

Attempt 3: Log to Database

Idea: store logs in SQL or NoSQL since it’s durable and queryable.

Sounds nice.

Until things break:

• DB inserts become bottleneck

• Locks + indexes kill write throughput

• Querying logs affects real app data

• Storage cost skyrockets

• Retention becomes a nightmare

A typical database can handle:

• MySQL: ~10k writes/sec (with tuning)

• MongoDB: 50k–200k writes/sec (clustered)

• Cassandra: high write throughput but still expensive at scale

1M/sec destroys it - financially and operationally.

Databases are for structured data, not firehose-scale telemetry.

Attempt 4: Send Logs Directly to Elasticsearch / OpenSearch

This is where many companies fail.

Elasticsearch gives:

• Searchability

• Visual dashboards

• Indexing

• Alerting

Feels perfect.

Until indexing overhead hits.

Indexing every event means:

• Tokenization

• Sharding

• Replication

• Compression

• Persistence

Even a large ES/OpenSearch cluster struggles beyond 200k–300k events/sec without: massive hardware , lots of tuning and big maintenance overhead

Lesson:
Elasticsearch/OpenSearch is great for querying logs - not ingesting raw firehose scale directly from apps.

Related [Before reading further]

This article scratched the surface. The real interview goes 10 levels deeper.

• How do you handle hot partitions?

• What if the cache goes down during a spike?

• How do you avoid counting the same view twice?

I’ve written an ebook that prepares you for all of it.

35 real problems. The patterns that solve them. The follow-ups you’ll actually face. The principles behind solving problems at scale, not just the final answers.

👉 Grab your copy here

The Scalable Architecture

To log 1M+ events/sec without hurting latency, you need one principle:

Logging must be asynchronous, streamed, and decoupled from the application.

Here’s the architecture that works:

1) Application Layer: Non-Blocking Logging

• Use async log appenders

• Logs go to in-memory ring buffers

• Logging never blocks request path

Libraries:

• Java: Logback AsyncAppender, Disruptor logging

• Go: Zerolog

• Node: Pino

• Rust: Tracing layer

No waiting. No flushing. No disk writes on main thread.

2) Queue or Stream: The Shock Absorber

This absorbs bursts and smooths throughput.

Use:

• Kafka

• Pulsar

• Kinesis

• Redpanda

• Google Pub/Sub

Benefits:

• Producers write at memory speed

• Consumers scale independently

• Backpressure is handled automatically

• Replay and retention available

Kafka alone can handle millions/sec with proper partitions.

3) Storage Layer: Hot + Warm + Cold

Different logs need different storage.

You don’t index everything - you promote only what’s useful.

4) Observability: Metrics Over Logs (When Possible)

Not every event needs a log line.

Sometimes counters, histograms, and traces are better.

Use:

• Prometheus

• OpenTelemetry

• Grafana Tempo

The mindset:

“Log what matters. Aggregate what repeats.”

Why This Architecture Works

This architecture works because logging no longer stands in the way of real work.

The app writes once and moves on - no waiting, no blocking. As traffic grows, you don’t fight the system, you just add more consumers and partitions.

Storage becomes smart instead of expensive: fast where needed, cheap everywhere else.

Even if parts fail, the system keeps absorbing events without dropping them. In the end, logs flow like a river - not a traffic jam.

Interview Answer

If asked:

How would you log 1M events/sec?

You respond:

I wouldn’t log directly. I’d log asynchronously, push events into an in-memory buffer, stream them through Kafka, and then store them in tiers: hot for recent searchable data, warm for analytics, and cold for long retention.

That way logging never slows requests, stays durable, and scales without rewriting the system.

Potential Follow Ups

• How to avoid log duplication?

• How to detect ingestion lag/remove backpressure?

• What if Kafka cluster goes down?

• How to avoid logging sensitive/PII data?

We’ll dive deeper into these kinds of follow-ups in upcoming articles - subscribe if you want more.

Thanks for reading. See you in next post.

Share

I still remember the first time I got an “Accepted” on CodeChef.

It was past midnight. My roommate was asleep, and I was hunched over a problem called “Chef and Subsequences.”
I’d been stuck for hours off-by-one errors, wrong answers on hidden test cases the usual pain.

When that green checkmark finally appeared, I didn’t celebrate loudly. I just leaned back, smiled, and felt that quiet satisfaction only programmers know the kind that says you earned it.

That was 2016.
Back then, DSA wasn’t a skill, it was a religion
You solved problems to prove you could think, not just code.

Fast forward to now.
I gave that same problem to ChatGPT and it solved it in 10 seconds.
Perfect output. Clean code. Explained every step like a teacher.

And I’ll be honest I didn’t feel proud or jealous.
I just felt… replaced.

The Autocomplete Era

Here’s what nobody tells you about AI: it didn’t just make coding easier. It made a specific type of intelligence feel obsolete.

You know that friend who memorized every LeetCode medium? The one who could implement Dijkstra’s algorithm on a napkin at lunch? Yeah, ChatGPT just made them irrelevant. The thing they spent 300 hours perfecting now comes free with a $20/month subscription.

It’s like training for years to be a human calculator, only to realize everyone has an iPhone.

Brutal? Maybe. But let’s be honest when was the last time you actually implemented a trie at work? I’ve been shipping code for ten years. I’ve used binary search twice. I’ve never once needed to color a graph outside of an interview.

So why are we still here, pretending DSA matters?

Because It Does - Just Not How You Think

Some time back, our notification feed imploded during peak hours. Users were staring at loading spinners for 3–4 seconds. Our support channel lit up. My manager sent a Slack message at 9 PM: “Fix this or we’re rolling back tomorrow.”

I opened the monitoring dashboard. Average response time: 2.8 seconds. Database queries: normal. CPU: fine. Memory: fine. What the hell?

Then I looked at the actual code. Some well-meaning engineer had added a “mark as read” feature. Sounds simple, right? Except we were checking if each notification was read by querying the database. For a user with 500 notifications, that’s 500 separate database calls. Every. Single. Time.

Classic N+1 query problem.

I didn’t need a fancy algorithm. I just loaded all the “read” notification IDs into a Set upfront one query instead of 500. Checked membership with notificationIds.has(id) instead of hitting the database each time.

Deployed it at midnight. Response times dropped to 180ms. Not perfect, but good enough that my manager stopped pinging me.

That’s not something you’d see on LeetCode. But it’s exactly what those hash table problems were training you for : recognizing when O(n) lookups are killing you and switching to O(1).

Here’s the thing: I could’ve asked ChatGPT to “optimize this feed endpoint.” It would’ve given me twelve different solutions Redis caching, query optimization, pagination improvements, database indexing. All technically correct.

But which one solves this specific problem without requiring a two-week refactor or convincing the infrastructure team to spin up a Redis cluster?

That’s the judgment call AI can’t make. It doesn’t know your codebase is held together with duct tape. It doesn’t know your team ships every Friday and breaking things now means working the weekend. It doesn’t know your database is already at 80% capacity and adding indexes might tip it over.

That intuition, knowing what’s actually feasible in your messy production environment, comes from pattern recognition you built grinding problems for years. You just didn’t realize you were learning how to think under constraints, not just how to code.

The Real DSA Lessons Nobody Talks About

Forget the algorithms for a second. What did grinding problems actually teach you?

Constraint thinking - Every problem on LeetCode starts with limits: “n ≤ 10⁵” or “must run in O(n log n).” Real systems have constraints too memory budgets, latency requirements, API rate limits. DSA taught you to start every problem by asking: “What can I not do?”

Trade-offs - Should you cache this? Pre-compute that? Use more memory to save CPU? These aren’t algorithm questions they’re engineering questions. But if you’ve never wrestled with time vs. space complexity, you’ll make expensive mistakes in production.

Debug instincts - When something breaks and your logs are useless, you need to think your way to the root cause. That’s the same skill you used debugging that wrong answer on Test Case 47. You just didn’t realize it was transferable.

I watched my teammate spend days tracking down a memory leak in our background job processor. He was convinced it was a database connection issue adding timeouts, tweaking pool sizes, restarting workers. Finally I looked at the code: he was loading 500k records into an array, processing them one by one, then loading another 500k. The array just kept growing. “Why not process in batches of 1000 and clear memory after each batch?” Twenty minutes later, memory usage was flat. He’d never thought about how data structures behave at scale.

That’s what happens when you skip the fundamentals.

What 2025 Actually Looks Like

Let me paint you a picture of the new reality:

You’re in a design meeting. Someone suggests using Redis for session management. Another person says, “Why not just use PostgreSQL with a TTL column?” AI can’t answer that. It’ll give you both implementations, beautifully documented. But you need to know that Redis is faster but costs more, that Postgres might bottleneck at scale, that your team doesn’t have Redis expertise.

Or imagine debugging a memory leak in production. AI will analyze your heap dumps and suggest fixes. But will you trust it blindfolded? Or do you need to understand why holding references to old objects prevents garbage collection?

The engineers crushing it in 2025 aren’t the ones with perfect LeetCode scores. They’re the ones who can:

• Look at a system design and smell the bottleneck before it happens

• Read AI-generated code and spot the edge case it missed

• Explain to a junior dev why we chose this approach over that one

DSA doesn’t make you better at writing code. It makes you better at thinking about code.

You don’t need to know how to implement a heap from scratch.

But you damn well better understand when a priority queue is the right tool, because that knowledge will save you when you’re optimizing a job scheduler before a product launch.

You don’t need to ace every Codeforces contest.

But if you can’t recognize that your “innovative new feature” is just a variation of the two-pointer pattern, you’re going to waste weeks reinventing wheels.

AI has commoditized syntax. It hasn’t commoditized judgment.

So What Now?

If you’re early in your career, yes grind some problems. Not because Google’s going to ask you to reverse a binary tree (though they might). But because it builds mental models you’ll use forever.

If you’re experienced, you already know: DSA isn’t your competitive advantage anymore. Your ability to apply those patterns to ambiguous, real-world problems is.

The future doesn’t belong to people who can reiterate algorithms. It belongs to people who can look at a pile of AI-generated solutions and know which one won’t fall apart under load.

AI may know all the answers.

But someone still needs to know which questions are worth asking.

And that someone better understand why hash maps exist.

Related

If you enjoy digging into how things actually work, I’ve been working on something larger alongside these articles.

I wrote an ebook that walks through 35 real engineering problems, focusing on how decisions break down at scale, what trade-offs actually matter, and the kinds of follow-up questions engineers run into in interviews and real systems.

It’s less about memorizing solutions, and more about learning the principles behind solving problems at scale - so you can reason through new situations, not just familiar ones.

👉 Grab your copy here

Thanks for reading. See you in the next post.

Share

Every modern product whether it’s a banking app, a ride-sharing platform, or a social network depends on one invisible yet powerful system: notifications.

They’re the tiny nudges that keep your product alive and your users informed.

• A push when your cab is 2 minutes away

• An SMS with your OTP

• An email confirming your order

Seems simple enough, right? Just send a message.

But here’s where the trap lies 👇

At small scale, everything feels easy. At scale, everything breaks.

For a prototype, you can literally loop over all users and call your provider’s API. Works fine for a few hundred users.
But when you need to send 100,000+ messages, “just send” quickly turns into “why is nothing working?”

Let’s see why.

Why This Problem Is Deceptively Hard

Imagine a banking app where 100,000 transactions fail due to a temporary outage.
You need to alert all affected users and fast.

What looks like a simple “send notifications” job now becomes a distributed system problem.

You’re suddenly fighting on multiple fronts:

• Throughput - Can you send thousands per second, not per hour?

• Reliability - What if FCM or SMTP is slow or returns 5xx errors?

• Idempotency - If your service crashes mid-way, can you retry safely without duplicates?

• Scalability - Will it still hold if you jump from 100k → 1M tomorrow?

• Observability - How do you know what succeeded, failed, or got delayed?

These five are the pillars every resilient notification system must solve.

First Attempts (and Why They Fail)

Approach 1: Loop and Send Directly

Idea: Just iterate through all users and send via Email/SMS/Push provider.

for user in users:
    sendNotification(user)

Why it feels right:
Simple. Easy to code. Works perfectly in dev.

Why it fails:
Each call might take ~300ms.
100k × 300ms = 30,000 seconds → 8+ hours!

You can try threading, but you’ll hit:

• Provider rate limits

• Connection errors

• Duplicate sends on retries

If your process crashes mid-way, you’ve no clue who got what.

Lesson: Code simplicity ≠ system simplicity. Throughput and fault tolerance don’t come free.

Approach 2: Use Database as a Queue

Idea: Insert all notifications in a table:

INSERT INTO notifications (user_id, status)
VALUES (user_id, ‘PENDING’)

A worker picks up PENDING, sends it, then marks SENT.

Why it feels right:
SQL is reliable. Transactions give confidence. You can count(*) to see progress.

Why it fails:
At 100k+, your DB becomes the bottleneck:

• Hot rows & lock contention

• Slow vacuuming

• Replication lag

DBs are great at state, not streaming events.
You’re forcing it to do something it wasn’t built for.

Lesson: Don’t turn your source of truth into a message queue.

Approach 3: Cron Job Batches

Idea: Run a job every minute that sends pending messages.

Why it feels right:
Scheduled jobs = easy mental model. “Send everything due now.”

Why it fails:
At the top of the minute → a burst of 100k requests.
Providers respond with 429 (rate-limit) or block your IP.
Your “real-time” notification now has up to 59s latency.

Lesson: Batching helps organization, not elasticity.

The Scalable Approach

To scale gracefully, separate concerns:

“Deciding what to send” ≠ “Actually sending it”

Think of it as an assembly line:

1. A producer decides what needs sending

2. A queue manages the flow

3. Workers deliver at safe speed

4. Metrics keep you informed

Let’s break it down.

1. Producer Service (Brain)

• Figures out who to notify (recipients)

• Prepares personalized payloads (Hello, Tom 👋)

• Publishes one message per user into a queue

It’s stateless: you can run multiple producers in parallel.
It doesn’t care when it’s delivered — just that it’s enqueued.

2. Queue or Stream (Buffer)

Use something like SQS, RabbitMQ, Kafka, Pub/Sub.

Why?

• Absorbs bursts — workers can pull at their own pace

• Handles retries gracefully

• Offers DLQ (Dead Letter Queue) for poison messages

• Enables backpressure — if providers slow down, queue length grows, not your CPU usage

Think of it as a shock absorber in your system.

3. Delivery Workers (Hands)

Each channel (push, email, SMS) has its own workers.

They:

• Pull messages at a controlled rate

• Enforce per-provider rate limits (avoid bans)

• Use idempotency keys (campaign_id + user_id + channel)

• Retry transient failures (e.g. network blips) with exponential backoff

• Send unrecoverable ones to DLQ for manual retry/inspection

Parallelize with more workers. They can scale horizontally.

4. Observability (Eyes)

You can’t fix what you can’t see.

Track:

• Metrics: enqueue/sent/failed/sec, retries, p95 latency

• Logs: idempotency key + provider response codes

• Alerts: spikes in DLQ, latency breaches, 5xx rates

Use tools like Prometheus + Grafana or Datadog.
Even a simple dashboard helps detect silent failures early.

Why This Architecture Works

Throughput:
50 workers × 35 req/s = ~1,750 req/s → 100k in ~1 minute

Reliability:
Queue buffers bursts, retries are idempotent.

Scalability:
Add workers; partition queues, scale independently.

Correctness:
Idempotency prevents duplicates, DLQ captures bad events.

Visibility:
Metrics & alerts show real-time health.

This is first-principles engineering:

• Decouple producers & consumers

• Isolate failures

• Scale horizontally

• Design for retries & observability

Interview Pivot (How to Explain It)

If asked in an interview 👇

“How would you design a system to send 100,000 notifications?”

You say:

“Naively, I’d loop through all users and call the provider directly but that couples expansion and delivery, and breaks with retries or rate limits.
Instead, I’d split concerns:
A producer expands recipients and enqueues messages.
Workers consume from the queue at a safe pace, enforce per-provider rate limits, use idempotency keys to prevent duplicates, and push failed messages to a DLQ.
That gives me high throughput, safety, and visibility 100k notifications in minutes, and scalable to millions.”

That’s not just an answer that’s a design mindset.

What’s Next?

We’ve laid the foundation - a robust, scalable notification system.
But there’s more to explore if you really want to think like an architect:

• How do we ensure each notification is sent exactly once?

• How do we handle sudden spikes, like 1 million events per second?

• What if iOS consumers stop working and fail to receive notifications?

• How do we alert engineers automatically when something breaks?

We’ll tackle these real-world challenges one by one in upcoming articles. Stay tuned.

Related

This article scratched the surface. The real interview goes 10 levels deeper.

• How do you handle hot partitions?

• What if the cache goes down during a spike?

• How do you avoid counting the same view twice?

I’ve written an ebook that prepares you for all of it.

35 real problems. The patterns that solve them. The follow-ups you’ll actually face. The principles behind solving problems at scale, not just the final answers.

👉 Grab your copy here

Thanks for reading. See you in next post.

Share

A few months back, I found myself staring at a problem that felt deceptively simple:
“We need a unified view of how our APIs are doing across the company.”

Sounds easy, right? Just throw some dashboards, grab a few metrics, and boom job done.
Except, no. This was one of those problems where the devil wasn’t in the details, the devil was the details.

Background

Our company had APIs everywhere - some powering core products, others quietly serving backend needs, and many more that just… existed. But no matter how important or obscure, they all had one thing in common:

Nobody had a single view of how they were doing.

• Were they following governance standards?

• Did they meet latency SLAs (P95, P99)?

• Were they failing silently somewhere?

Each team had their own dashboards, their own logs, their own way of tracking. Imagine trying to run a city where every neighborhood has its own maps, traffic lights, and bylaws but nobody talks to each other. That’s where we were.

Our mission: build a “control tower” for all APIs in one place.

The Naive First Answer: “Just Read the Logs”

The obvious first thought was: well, the data already exists. APIs log everything - successes, failures, latencies, requests. Just scoop it up.

Then we asked our edge gateways what they were logging, the answer came back like a punch:

20–25 terabytes of logs per day. Billions of events.

Let that sink in. Every. Single. Day.

That meant:

• Gigantic ingestion pipelines.

• Massive storage costs (cold + hot tiers).

• Complex transformation logic before we could even query anything useful.

What sounded like “just read the logs” turned into “build a data warehouse project the size of a small startup.”

Talking to the Gatekeepers

The next step was obvious but painful: talk to the teams running these edge gateways.

This is where the non-technical pain starts:

• They spoke a different “language” than us - security and infra-first, while we cared about API-level insights.

• Their logs had dozens of fields we didn’t care about, and were missing fields we desperately needed.

• They already had their workflows; our requests felt like interruptions.

We had to set up meeting after meeting:

• “What exactly are you logging?”

• “Can we get field X?”

• “What does this cryptic field mean?”

• “Oh, you’re logging to Splunk - can we tap into it?”

And yes, Splunk had all the data. But it wasn’t designed for the kind of structured, API-centric queries we wanted.

The Pain of Building From Scratch

At this point, the obvious engineering option was:

• Ingest logs from Splunk.

• Store them in S3 or Databricks.

• Build structured tables.

• Layer insights pipelines on top.

It sounds like progress… but think about the effort:

• Time: months just to get ingestion stable.

• Cost: storing petabytes within weeks.

• Ops Overhead: keeping pipelines alive against daily terabytes.

• Redundancy: building yet another giant data system when the company already had dozens.

It felt wrong. Painfully wrong.

The Twist: Someone Already Did the Hard Part

Here’s where luck and persistence collide.

While poking around, we discovered another team in the company had already built a massive ingestion pipeline - for security events.

• They were reading from 60+ different log sources.

• Normalizing data into the OCSF format (Standard Logging).

• Storing everything in structured tables.

• Running Kafka, databases, pipelines - the whole machinery.

They weren’t solving our problem, but the foundation was eerily similar.

So we had a choice:

• Option A: Keep building our own “parallel empire,” doubling costs and effort.

• Option B: Swallow our ego, walk over, and ask to collaborate.

We chose Option B.

Collaboration ≠ Easy

Now, collaboration sounds noble in principle. But in practice, it’s one of the hardest things in engineering.

• Their priorities were security, ours were API observability.

• Their timelines were different.

• Their formats weren’t built with APIs in mind.

• And, let’s be honest, not everyone likes outsiders coming in with “we want to use your system for our stuff.”

It took weeks - literally three to four weeks - of discussions, clarifications, requirement docs, and alignment calls.

Some conversations went in circles:

• “Can we add these fields?”

• “Will your pipeline support our query scale?”

• “How much additional load will this put on you?”

At times, it felt like more diplomacy than engineering.

The 10% Coding Myth

Here’s the part that might shock a lot of junior engineers: the actual coding part of this project was maybe 10–15% of the total effort.

The rest?

• Negotiating requirements.

• Understanding existing systems.

• Aligning with teams who didn’t always share our goals.

• Making trade-offs with cost, scale, and compliance.

• Documenting and explaining why we weren’t reinventing the wheel.

And this is the pain nobody prepares you for: the pain of alignment.

Where AI Still Fails

People often ask: “Why won’t AI replace engineers soon?”

Here’s why:

AI can generate SQL queries, code snippets, maybe even pipeline scripts. But it can’t:

• Sit in a room with five stakeholders and resolve conflicting goals.

• Decide whether to build vs reuse based on organizational context.

• Balance storage costs against real-time needs.

• Handle the messy human side of building systems inside large organizations.

That’s the stuff that eats up months and it’s the stuff AI can’t touch (yet).

The Endgame

Eventually, we piggybacked on the existing ingestion pipeline.

We stored billions of events in cold storage, moved slices into hot storage for fast queries, and built insights pipelines on top.

The result: a working API observability layer without rebuilding the entire ingestion empire from scratch.

But the lesson wasn’t just about logs, APIs, or storage tiers.

Engineering is not just:

• Knowing the best data structure.

• Writing the cleanest code.

• Or deploying the fastest service.

It’s about solving problems inside the real-world mess:

• Systems already exist, often imperfectly.

• People have competing goals.

• Costs and trade-offs matter.

• Reinventing the wheel is easy but dangerous.

Engineering = 20% tech, 80% alignment.

That’s the truth no coding tutorial tells you.

When I look back, the story wasn’t about APIs at all. It was about realizing that as you grow in your career, the bottleneck isn’t your coding speed. It’s your ability to navigate systems - technical, organizational, and human.

And if you ever find yourself drowning in 25 terabytes of logs a day, remember: the solution might not be another line of code. It might be a conversation.

We want to let users upload a video and let millions of other users watch it smoothly.
That sounds simple, but videos are huge (hundreds of MBs), networks are unreliable, and viewers expect instant playback.

So our design has to solve:

• Storing large video files safely.

• Converting them so they play well on all devices.

• Delivering them fast worldwide.

• Scaling as traffic grows from 10 users to millions.

Minimum Viable Product (MVP)

Let’s start with the simplest working version.

Components

Client Upload → Object Storage

• Instead of sending a huge file through our server, the client uploads directly to Object Storage (like Amazon S3 or Google Cloud Storage).

• Why? Because it’s built to store huge files, automatically replicated, and cheap.

App Server (Metadata)

• Handles small data: title, description, user info, publish status.

• Stores this in a Relational Database (Postgres/MySQL).

Transcoding Service (Video Processor)

• Takes the uploaded raw video and converts it into multiple formats (240p, 480p, 720p).

• Why? Different devices and network speeds → different quality needed.

CDN (Content Delivery Network)

• A global network of servers that caches video segments close to users.

• Why? Without CDN, every play would hit our storage directly → slow and expensive.

Queue (Task Manager)

• When a video is uploaded, we push a “transcode this file” job into a queue.

• Workers pull jobs one by one → keeps system stable even if thousands of uploads happen.

Database Glimpse

-- Who uploaded what
CREATE TABLE users (
  id BIGINT PRIMARY KEY,
  name VARCHAR(100),
  email VARCHAR(200) UNIQUE
);

-- Core video table
CREATE TABLE videos (
  id BIGINT PRIMARY KEY,
  owner_id BIGINT REFERENCES users(id),
  title VARCHAR(200),
  description TEXT,
  status VARCHAR(20),   -- uploaded, processing, published
  created_at TIMESTAMP
);

-- Which renditions exist
CREATE TABLE renditions (
  video_id BIGINT REFERENCES videos(id),
  quality VARCHAR(10),  -- 240p, 480p, 720p
  path TEXT,            -- storage/CDN path
  PRIMARY KEY(video_id, quality)
);

Why This Works (MVP)

• Simple: only 5 moving parts.

• Resilient: queue prevents overload.

• Cost-efficient: CDN cuts bandwidth bills, storage is cheap.

But… it’s limited:

• Startup latency (may take 2–3 seconds to buffer).

• Limited quality options.

• All running in one region → bad for global users.

Scaling Up (When Traffic Grows)

When our MVP gets popular (say, 100k uploads/day, 1M viewers), cracks appear. Let’s evolve step by step.

Playback Experience

Issue: Viewers wait 2–3 seconds before playback begins.

Solution: Preload the first few seconds into the CDN as soon as a video is published. Playback feels instant.

Upgrade: Add higher qualities like 1080p, but only for videos that cross a popularity threshold. This saves storage and processing costs.

Database Load

Issue: With millions of requests, the relational database struggles with queries like trending videos, likes, and search.

Solution:

• Add a cache (Redis) for popular data.

• Use a search index (Elasticsearch) for fast lookups.

-- Daily view counts (approximate is fine)
CREATE TABLE video_views (
  video_id BIGINT REFERENCES videos(id),
  day DATE,
  approx_count BIGINT,
  PRIMARY KEY(video_id, day)
);

Exact counters aren’t necessary; eventual consistency is good enough for user experience.

Cost Control

Issue: Storage and bandwidth bills grow rapidly.

Solution:

• Move older, rarely watched videos to cold storage.

• Use per-title encoding to compress intelligently, cutting bandwidth by 20–40%.

Global Expansion

Issue: A viewer in UK fetching video data from a US server faces high latency.

Solution: Replicate storage across regions and let CDNs serve videos from the nearest origin.

Reliability and Failures

Issue: Transcoding jobs pile up or storage has a temporary outage.

Solution:

• Use dead-letter queues to handle failed jobs.

• Autoscale worker pools; temporarily skip highest quality and backfill later.

• Multi-region storage ensures redundancy.

Lessons from YouTube’s Growth

From a single bucket and database, the system grows into a global platform:

We started with the simplest version of YouTube: a bucket for files, a database for metadata, and a queue for processing. That was enough for the first few thousand users.

But real systems don’t stay small. As traffic grew, new challenges appeared one by one: buffering, database overload, rising costs, global latency, and reliability concerns. Each time, the solution wasn’t magic it was applying first principles:

• Keep data close to users.

• Optimize for real-world networks.

• Spend compute and storage only where it creates visible value.

• Design for failure so the system bends instead of breaking.

The end result is what we see today: a system that can absorb millions of uploads and serve billions of daily views without users thinking twice about what happens behind the scenes.

System design isn’t about memorizing complex diagrams it’s about starting with a clear mental model, asking “what must always be true?”, and evolving step by step as scale demands.

Share

In the early 2010s, MongoDB was everywhere.
Meetups, blog posts, hackathons everyone was saying the same thing:
Forget SQL. Just store JSON and ship faster.

And for a certain kind of project, MongoDB was a breath of fresh air. No schemas, easy to install, and “web scale” (a phrase that became a meme after that infamous video).

But hype doesn’t last forever.

What Problem Did MongoDB Solve?

Back then:

• Startups wanted to build fast without getting bogged down in database schema design.

• Developers hated ALTER TABLE migrations just to add a single field.

• The world was moving toward JSON APIs - storing JSON directly made sense.

MongoDB gave you:

• Schema-less storage : Add a new field to your documents without a migration.

• Developer-friendly : Query with a syntax that felt closer to JavaScript than SQL.

• Horizontal scaling : Shard your data across servers to handle large workloads.

A 2011 social media startup could ship features quickly - today you store { name, bio }, tomorrow you add { profilePic } - without touching migrations or breaking existing data.

Where MongoDB Fell Short

The marketing was great. The reality was… mixed.

1. Schema-less ≠ Structure-less
   “No schema” often turned into “no consistency.” You’d end up with {name} in some documents, {fullName} in others, and {n} in a few because someone fat-fingered it.

2. Joins? What Joins?
   MongoDB didn’t do joins well (or at all in early days). If your data was relational - orders, customers, products - you either duplicated data everywhere or wrote ugly aggregation pipelines.

3. Early Durability Issues
   Older MongoDB versions had cases where writes could be acknowledged before hitting disk. That’s not great if you care about not losing money or orders.

4. Sharding Complexity
   Yes, you could scale horizontally - but running and maintaining a sharded Mongo cluster was an ops headache.

How Modern Solutions Beat or Complement It

MongoDB isn’t “dead” like Hadoop hype - it’s still widely used - but it’s no longer the “SQL killer” it was marketed as.

1. Postgres Got JSON

• PostgreSQL added JSON/JSONB columns with indexing, giving you the flexibility of Mongo plus relational power.

• You can store semi-structured data and still run SQL joins when you need them.

2. Managed Mongo (Atlas)

• Atlas fixed some pain points but it’s basically Mongo with training wheels. You’re still dealing with document design pitfalls if you misuse it.

3. Better Fit-for-Purpose Choices

• DynamoDB for massive key-value workloads.

• Elasticsearch for text search.

• Redis for caching.

A Practical Comparison

Old Way (Mongo, 2012):

• Launch MVP fast.

• Add fields freely without migrations.

• Denormalize data for performance.

• Regret it later when analytics and reporting need consistency.

New Way (Postgres with JSONB, 2025):

• Use strict schemas where you need relational guarantees.

• Store flexible JSON for fields that change often.

• Keep queries consistent, indexed, and optimized.

Example

MongoDB’s Original Pitch : "Need to store flexible data like user profiles with different fields for each user? MongoDB lets you add { favoriteFood } to one document and { petName } to another without schema changes."

Now (Postgres with JSONB):
JSONB column: store flexible fields inside a relational table.

• Need favoriteFood for one row? Add it in JSONB.

• Still join on user_id like a normal SQL table.

• Create an index on favoriteFood if needed.

The only reason many teams picked Mongo flexible documents is now built into Postgres, but without sacrificing joins, ACID, or query power.

Where MongoDB Still Works Well

• Rapid prototyping where schema changes daily.

• Event logging or analytics where documents don’t need complex joins.

• Content-driven apps (CMS, catalogs) where items are self-contained.

MongoDB wasn’t a SQL killer. It was a shortcut for certain workloads that sometimes turned into technical debt.
Used with discipline, it’s still a solid choice. But if you treat “NoSQL” as “No Rules,” you’ll end up with a data swamp.

The NoSQL revolution didn’t kill SQL it just made Postgres even stronger.

Share

Building a product that scales from a handful of users to millions isn’t about pulling some magic architecture out of thin air. It’s about knowing what really matters at each stage and not wasting time on stuff you don’t need yet.

This article breaks down the journey from 0 to1M users step by step, what to focus on, what tech choices make sense, and when to level up your stack.

0 to 100 Users: Just Ship It

At this stage, your goal isn’t scale, it’s speed of development. You want to validate the idea, get feedback, and iterate fast. Over-optimizing early wastes time and slows you down.

Tech Requirements

Infrastructure

• Single server (monolith): One machine running everything : app, database, and static assets.

• Keep it simple to deploy and easy to debug.

Database

• One relational database like PostgreSQL or MySQL.

• No need for sharding, replicas, or complex schemas yet.

• Focus on correctness and easy migrations.

Authentication

• JWTs or local sessions.

• Lightweight and easy to implement.

• Don’t waste time building complex auth flows unless core to your app.

Monitoring

• At this point, optional. Basic server uptime monitoring is enough.

What to Focus On

• Speed of Development: Get features out quickly, gather user feedback, iterate fast.

• Keep It Maintainable: Even in MVP, write code that you or your co-founder can understand a month later.

• Avoid Premature Optimization: Don’t worry about caching, load balancing, or auto-scaling yet.

At <100 users, bottlenecks are rarely technical. The real challenge is finding product–market fit. Running lean helps you test hypotheses quickly without being bogged down in infrastructure.

100 to 1,000 Users: Clean It Up

Now that people are actually using your service regularly, small cracks in your MVP start to show : slow queries, messy code, occasional downtime. This stage is about introducing stability and preparing for real growth.

Tech Requirements

Code

• Add modular layers: break spaghetti code into services/routes/components.

• Set up a clean folder structure for controllers, services, and utilities.

• Add basic testing (unit/integration).

Database

• Start indexing frequently queried columns.

• Apply basic query optimization to prevent N+1 queries and unnecessary full scans.

• Introduce migration scripts if you haven’t already.

Infrastructure

• Add NGINX or a simple load balancer in front of your app.

• This helps distribute load if you start running multiple app instances.

Monitoring

• Implement basic logs and uptime checks.

• Tools: Papertrail, Datadog (lite), or even a cron job with cURL + Slack alerts.

• You don’t need full observability yet, but you need visibility when things go down.

Authentication

• If you started with JWT/local, ensure token refresh flows and secure session management.

• Add rate limiting for login attempts (prevent brute force).

What to Focus On

• Stability: Keep the app reliable as traffic grows.

• First Signs of Load: Start watching for DB slowdowns, memory spikes, or CPU bottlenecks.

• User Trust: Downtime hurts more now because people rely on your product daily.

At 100–1000 users, the infrastructure still doesn’t justify Kubernetes clusters or multi-region databases. Instead, the biggest wins come from low-hanging fruit like indexes, load balancing, and monitoring.

This is usually the stage where founders are tempted to over-engineer (“let’s move to microservices!”).

Don’t.

Stick to a monolith, but make it clean and testable.

1,000 to 10,000 Users: Prepare to Scale

At this level, you can’t rely on quick hacks. You’ll start to feel database strain, server latency, and the risk of downtime. This stage is all about reducing bottlenecks before they crush you.

Tech Requirements

Infrastructure

• Move to the cloud (AWS EC2, GCP GKE, or Azure equivalents).

• Introduce containerization (Docker) for consistency between dev and prod.

• Possibly multiple app instances behind a load balancer.

Database

• Implement connection pooling (e.g., PgBouncer for Postgres).

• Add read replicas to offload reporting/analytics queries.

• Monitor slow queries regularly.

Cache

• Add Redis or Memcached for hot data (user sessions, popular queries).

• This prevents your database from melting under repetitive reads.

Async Processing

• Introduce background workers/queues (Celery, Sidekiq, or RabbitMQ).

• Handle tasks like sending emails, notifications, report generation, etc. outside of request-response cycle.

• Keeps user-facing requests fast.

Monitoring

• Move beyond uptime checks, start application-level monitoring.

• Tools: Prometheus, Grafana, Datadog, or New Relic (depending on budget).

What to Focus On

• Reducing Latency: Every millisecond counts when thousands of requests hit.

• Preventing DB Overload: Without pooling, caching, and replicas, DB will become your bottleneck.

• Stability Under Load: Ensure background jobs and caching absorb spikes gracefully.

This stage isn’t about reinventing architecture yet it’s about reinforcing your monolith with strong supports.

• Most downtime at this level is caused by database saturation or blocking tasks hogging threads.

• By adding caching + queues, you essentially buy yourself more runway before needing a distributed architecture.

You’ll feel tempted to split into microservices here. Resist unless absolutely necessary. A well-structured monolith with caching and async workers will handle 10k users just fine.

10,000 to 100,000 Users: Go Distributed

At this stage, your single beefy monolith (with caching and workers) will start showing cracks. Request volumes spike, features multiply, and downtime gets very expensive. You now need to distribute responsibility across systems for performance and reliability.

Tech Requirements

Services

Begin splitting the monolith into smaller services.

• Start with high-load components: authentication, notifications, file uploads.

• Keep the “core” still monolithic for now to reduce complexity.

Scaling

• Add auto-scaling rules for app servers (scale horizontally during traffic spikes).

• Use a CDN for static content (images, CSS, JS) to offload delivery and reduce latency worldwide.

Queueing

• Implement event-driven processing with Kafka, RabbitMQ, or AWS SQS.

• Queue systems help decouple heavy tasks (e.g., analytics, billing) from user-facing requests.

Monitoring

Move to advanced monitoring + alerting.

• Metrics: Prometheus + Grafana, Datadog, or similar.

• Add error tracking (Sentry, Rollbar).

• Track latency, throughput, and error rates across services.

Database

Likely still one main DB, but:

• Use read replicas heavily.

• Start planning for partitioning or sharding if write load grows.

• Add backup & recovery automation.

Infrastructure

• Leverage orchestration (e.g., Kubernetes, ECS) for managing multiple services/containers.

• Introduce service discovery (so microservices can find each other).

What to Focus On

• Reliability Under Pressure: Users now expect “always-on” performance.

• Performance: Scaling horizontally and caching smartly to handle spikes.

• Error Visibility: You need to know about problems before your users do.

At this level, performance issues compound:

• A single DB query running slow now impacts thousands.

• One service crashing can bring down the system if it’s not isolated.

• Event queues + CDNs + monitoring make the system resilient.

Don’t rush into a full microservices architecture. Instead, extract only the pain points into separate services. A carefully modularized system scales far better than a chaotic microservice.

100,000 to 1M Users: Serious Scale

By now, you’re not just running a product, you’re operating a distributed system that needs cost efficiency, fault tolerance, and global reach. Every inefficiency, every DB bottleneck, every network hiccup gets amplified at scale.

Tech Requirements

Database

• Sharding: Split data across multiple DB instances to handle writes.

• Denormalization: Optimize queries by storing precomputed or duplicated data when needed.

• Time-series storage (for metrics, logs, analytics).

• Ensure automated failover + backups across regions.

Cache

Implement multi-layer caching:

• Client → CDN → Edge cache → Application cache → DB

• E.g., Cloudflare/Akamai at the edge, Redis/Memcached at the app layer.

Goal: keep hot paths away from the DB entirely.

Infrastructure

• Global load balancers to route users to the nearest healthy region.

• Traffic routing (GeoDNS, Anycast) for latency optimization.

• Add edge functions (e.g., Cloudflare Workers, AWS Lambda@Edge) for things like authentication checks or lightweight personalization close to users.

Compliance & Security

• Enforce audit logging across all sensitive actions.

• Set up rate limiting and bot protection at the edge.

• Define clear security policies (SOC 2, GDPR, HIPAA if relevant).

Monitoring & Observability

• Full observability stack (metrics, logs, traces).

• Distributed tracing (Jaeger, Zipkin, OpenTelemetry).

• Automated error recovery scripts (self-healing).

DevOps Practices

• CI/CD pipelines with canary deployments.

• Blue-green or rolling deploys to minimize downtime.

• Chaos engineering (Netflix-style) to test failure scenarios.

What to Focus On

• Cost Optimization: infra bills will skyrocket, caching/CDNs reduce costs.

• Failover & Resilience: one region going down shouldn’t kill your app.

• Global Latency: keeping response times low worldwide.

At 100k–1M users, your challenges shift from “will this hold up?” to “can we keep it cost-effective, secure, and globally fast?”

• Sharding and caching prevent DB meltdowns.

• Multi-region load balancing keeps uptime near 100%.

• Compliance and audit logging protect you from business-ending mistakes.

Think of this stage less like coding an app and more like running an airline: uptime, safety, and efficiency are the priorities. Features matter, but reliability is the brand now.

Scaling to a million users isn’t a single “big bang” move, it’s a series of practical upgrades at the right time.

The biggest mistake most teams make?

Either over-engineering too early or waiting too long to fix cracks.

If you focus on the right things at the right stage, you’ll not only handle the load: you’ll stay sane

Share

Writing tests isn’t just about coverage it’s about confidence.
Here are 5 lesser-known but powerful habits to make your unit tests rock-solid and readable.

1) Test behavior, not implementation

Write tests that check what the code should do - not how it does it internally.

Internal methods may change, but expected behavior should remain consistent.

Bad Example (testing implementation):

@Test
void shouldCallApplyDiscountMethod() {
    PricingService spy = Mockito.spy(new PricingService());
    spy.getPrice("premium", 100);
    verify(spy).applyDiscount();  // you're testing internal method call
}

This test will fail if you refactor and remove applyDiscount(), even if the behavior (discounted price) remains correct.

Good Example (testing behavior)

@Test
void shouldReturnDiscountedPriceForPremiumUser() {
    double price = pricingService.getPrice("premium", 100);
    assertEquals(80, price);  // checks outcome, not method internals
}

This test will still pass even if you later use a different way to calculate discounts.

Test what your users care about - not the plumbing behind it.

2) Name tests like documentation, not code

Write test method names like they’re telling a story - clear, specific, and human-readable.

It will:

• Makes test reports easy to scan - no need to dig into code.

• Helps future devs understand intent instantly.

• Acts like living documentation for edge cases and requirements.

Bad Example (vague, codey)

@Test
void testLoginFail() {
    // What kind of fail? Why? What user? No clue.
}

Good Example (self-explaining):

@Test
void shouldThrowExceptionWhenPasswordIsInvalid() {
    assertThrows(AuthException.class, () -> authService.login("john", "wrongpass"));
}

Without even reading the code, you know the exact scenario being tested.

Well-named tests read like user stories - not cryptic code snippets.

3) Write the test before fixing the bug

When you discover a bug, write a test that reproduces it before fixing the code.

Why?

• It proves the bug actually exists (not just in your head).

• When you fix the code, the test passes, now you know the bug is fixed properly.

• If someone reintroduces that bug later, the test will catch it.

Let’s say you find that leap year birthdays are crashing your system.

// Write the test first — it fails right now.
@Test
void shouldAllowLeapYearBirthday() {
    User user = userService.register("John", LocalDate.of(2024, 2, 29));
    assertNotNull(user);
}

You run it - it throws an exception or fails.
Now fix the logic in userService.register().
Once fixed, the test passes. Done.

A failing test written before the fix proves the bug and guards against its comeback.

4) Use data-driven tests for edge cases

Instead of writing many similar tests, loop through a set of inputs using parameterized tests.

It prevents copy-paste clutter and makes test suite cleaner.

Also edge cases are easier to discover when grouped together.

Bad Example (repetitive and bloated):

@Test
void shouldRejectEmptyUsername() {
    assertFalse(userValidator.isValidUsername(""));
}

@Test
void shouldRejectSpaceOnlyUsername() {
    assertFalse(userValidator.isValidUsername(" "));
}

@Test
void shouldRejectNewlineUsername() {
    assertFalse(userValidator.isValidUsername("\n"));
}

Good Example (clean and scalable):

@ParameterizedTest
@ValueSource(strings = {"", " ", "\n"})
void shouldRejectInvalidUsernames(String input) {
    assertFalse(userValidator.isValidUsername(input));
}

One test, multiple edge cases covered and easy to add more later.

Edge cases love company - test them together with parameterized inputs.

5. Assert what matters, not every field

Focus your assertions only on the important business outcomes, not on every field of the object.

Why?

• Keeps tests stable even as non-essential details evolve.

• Makes your intent crystal clear: you’re testing this, not everything.

• Saves time in test maintenance during refactors.

Bad Example (over-asserting, brittle):

@Test
void shouldCreateOrderSuccessfully() {
    Order order = orderService.create(user, cart);

    assertEquals(123, order.getId());
    assertEquals("2024-07-19", order.getCreatedAt().toString());
    assertEquals("John", order.getCustomerName());
    assertEquals(2500, order.getTotal());
}

This test breaks if ID generation changes, date format updates, or name field is renamed - even if total amount logic is perfect.

Good Example (assert business-critical value):

@Test
void shouldCalculateTotalCorrectlyInNewOrder() {
    Order order = orderService.create(user, cart);
    assertEquals(2500, order.getTotal()); // This is what actually matters
}

Now the test will only fail if the real logic (total amount) breaks.

Test the contract, not the cosmetics, assert what users or business actually care about.

Most developers write tests. Great developers write tests that are useful.
Adopt even one of these tips, and you (and teammates) will notice the benefit.

Designing scalable systems isn’t just about knowing the tools it’s about asking the right questions upfront. Before you look into architecture diagrams, here’s a checklist every developer should walk through.

1. Understand the Goal

Before writing a single line of code or diagramming a system, ask:

• What exactly are you building? (e.g., a social media platform)

• Who’s using it? (teenagers, professionals, businesses)

• Main use case: (sharing photos, professional networking, online payments)

Without a clear goal, you may end up overengineering, solving the wrong problem, or missing critical user expectations.

A system for internal use by employees has vastly different needs than one for millions of external users.

2. Traffic Pattern: Read-Heavy vs Write-Heavy

This helps you determine where to focus your optimization efforts.

• Systems like Netflix prioritize fast content delivery. Caching, CDN, and read replicas are essential.

• Apps like WhatsApp focus on handling huge volumes of incoming data reliably, using queues and write buffers.

Designing a read-heavy system with write-heavy optimizations leads to wasted resources and poor performance. This is one of the most foundational choices in backend architecture.

3. Consistency vs Availability

In distributed systems, you often can’t have both perfect consistency and high availability (CAP theorem).

• Strict consistency: Needed when any mismatch is unacceptable (e.g., showing $500 when your bank balance is actually $300 is a big problem).

• Eventual consistency: Perfectly fine when real-time precision isn’t required (e.g., social feeds showing a post a few seconds late).

Misjudging this can lead to system unreliability or over-complexity. Choose based on business risk.

4. Latency Requirements

Do users expect instant results, or can they wait?

Real-time (low latency): Critical for gaming, live streaming, or payment systems. Use caching or precomputed data.

• e.g Think of Zoom. A 1-second delay ruins the experience or a Google Pay transactions.

Async (higher latency is okay): Tasks like email delivery or report generation.

• e.g Email confirmations can take a few seconds without hurting UX, analytics reports generation.

This directly influences your use of technologies like message queues, caching, or background workers. Over-optimizing for real-time needs can make your system costly and fragile.

5. Expected Scale

Think long-term. You might start with 100 users - but will it still work with 10 million?

Initial design (MVP) can be simple but should easily scale.

• Google Docs: Started small, now supports millions of users editing simultaneously.

• Uber: Handles thousands of ride requests per minute with horizontal scaling and stateless services.

Systems often fail not because of bugs, but because they weren’t designed to handle growth. Google Docs scaled from a basic collaborative editor to a complex real-time system. If you don’t design with scalability in mind, rewrites become painful and expensive.

6. Access Patterns

Identify what data is accessed most frequently or in which pattern.

If you choose the wrong pattern (e.g., hitting DB for every search), you risk slow performance, high cost, and poor UX. Patterns should align with actual user interaction and scale expectations.

7. Data Growth & Partitioning

Data will grow. Will your system crash when it does?

• YouTube Comments → Sharded by Video ID
  Each video can have millions of comments, but comments under different videos are unrelated. So YouTube shards the data by video_id to avoid one giant table and to make reads/writes faster.

• Twitter Tweets → Sharded by User ID + Time
  Tweets are partitioned by user_id and sometimes by time buckets. This allows fast access to a user's timeline and prevents hot partitions when a user tweets frequently.

• Bank Transactions → Partitioned by Account Number + Date
  To retrieve and archive transactions efficiently, banks shard data by account number and archive old records beyond 6–12 months to cold storage (e.g., tape drives or long-term S3).

• Google Drive / Gmail → Archive Cold Data
  Old files or emails are rarely accessed but take up space. These are moved to low-cost, long-term storage, improving DB performance and cutting storage costs.

As your app scales, unpartitioned data becomes a bottleneck. Without planning for growth, even a basic query like “get all orders for user X” can slow down, affecting everyone.

8. Failure Handling

Things break. Servers crash. Networks fail. How does your system cope?

You don’t want users facing 500 errors every time something small fails. Proper retries, fallbacks, and circuit breakers ensure graceful degradation and better user trust.

9. Security & Auth

If your app is public-facing or handles user data, you must think about:

• Who can access what?

• How do you prevent abuse?

Security is often treated as an afterthought — until something goes wrong. Implementing OAuth, rate limiting, and CAPTCHA from day one protects your system and your users.

10. Do You Need to Build It All?

You don’t need to reinvent the wheel. Use existing, battle-tested services.

Can you use a 3rd party for auth, payments, media, etc.?

Offload non-core features to SaaS or APIs

Building everything in-house is tempting but inefficient. Offloading non-core parts lets you focus on your unique value. If Uber built its own media service or SMS gateway from scratch, it would have slowed their growth drastically.

These aren’t hard rules - but they’ll save you from hard rewrites. Think of them as your system design compass.

And once your system is built, don’t forget to think about observability (logs, metrics, alerts), deployment strategy (CI/CD, rollback), data backups, cost optimization, and team ownership.

Because building is just the beginning - running it well is where the real game begins. Will share that in some other article.

When your logs say everything… and nothing.

Almost every developer has done it:

for (Order order : orders) {
    log.info("Processing order {}", order.getId());
    process(order);
}

It feels harmless when you have ten test orders on your laptop.
Push the same code to production - where there are two million orders an hour and suddenly:

• Pods get evicted because log files fill the disk.

• Your observability bill shoots past your coffee budget.

• Real errors disappear under a waterfall of “Processing order…”.

This is what happens when we use logging as a substitute for design.

Logging isn’t the villain. Using it as duct-tape for deeper design problems is.

Below are four everyday logging mistakes (I’ve made every one of them), the pain they cause, and simple ways to turn the noise into real insight.

1. Hot-Loop Logging

for (User u : userList) {
    log.debug("User found: {}", u.getEmail());
    sendWelcomeMail(u);
}

Why it stings

• CPU & I/O drain: Formatting strings and writing to disk thousands of times a second steals cycles from business logic.

• Real-world fallout: One team I coached saw their Kubernetes node hit “DiskPressure” after 40 minutes of a bulk import - logs alone chewed 22 GB.

• Debugging blindfold: When everything is logged, nothing stands out.

Fix in five minutes

int processed = 0;
long start = System.nanoTime();

for (User u : userList) {
    sendWelcomeMail(u);
    processed++;
}

log.info("Sent {} welcome mails in {} ms",
         processed,
         (System.nanoTime() - start)/1_000_000);

Log once, summarise, move on.
Need per-user data? Emit a metric (welcome_mail_total) and graph it, no human wants to read 100 000 log lines about it.

2. Boolean-Flag Noise

log.debug("isActive=" + user.isActive());

Even when DEBUG is disabled, the JVM still builds that string. Repeat in a hot path and the garbage collector becomes your shadow.

Fix in five seconds

if (log.isDebugEnabled()) {
    log.debug("isActive={}", user.isActive());
}

• The guard stops needless work.

• The {} placeholder defers formatting until the line is actually written.

3. Panic-Level Misuse

log.error("Invalid coupon code: {}", code);

A user typo is annoying, not an outage, yet it pages the on-call engineer at midnight.

How to choose the right level

Route ERROR to PagerDuty, send WARN to a Slack channel, and let INFO live quietly in your log store.

4. “Printf Metrics”

long t0 = System.nanoTime();
// DB query…
log.info("query_ms={}", (System.nanoTime() - t0) / 1_000_000);

You later write a regex in Grafana to graph the numbers. One day someone changes the log text, the regex breaks, and alerts go silent.

Better: emit a real metric

Timer queryTimer = Metrics.timer("db.query.duration");
long t0 = System.nanoTime();

runQuery(sql);

queryTimer.record(System.nanoTime() - t0, TimeUnit.NANOSECONDS);

• Percentiles, alerts, and dashboards stay intact no matter how the log message evolves.

• Logs tell stories; metrics track numbers; traces show relationships. Keep the channels separate.

Choosing Not to Log

A Simple Logging Playbook You Can Adopt Tomorrow

• Write a one-page team policy
  Levels, PII rules, retention. Consistency beats “whatever the last person did”.

• Set a daily log budget
  Choose a number (e.g., “1 GB per service per day”) and alert when you get close.

• Add request IDs automatically
  With SLF4J’s MDC or Log4j ThreadContext. One-click correlation beats grep.

• Batch or sample in bulk jobs
  Log the first record, every thousandth, and the last. Or just the summary.

• Review the top 10 chatty log statements every sprint
  Deleting dead noise is the fastest performance win you’ll ever get.

Logging is like salt in cooking: essential, but easy to overdo. Whenever you add a line, ask yourself:

1. Who will read this and why?

2. Is there a cheaper/faster way to get the same insight?

3. Could this message become dangerous at scale?

Answer those three and you’ll log just enough and probably sleep better on-call.

Share

Writing correct code is only step one.
Keeping it alive, fast, and resilient in production is a different game.
These lessons aren’t just theory - they’re the hard truths that every backend developer learns after their first real outage, rollback, or midnight PagerDuty call.

Read them before production teaches them to you the hard way.

1. Production readiness involves more than code correctness

Just because your code compiles and passes tests doesn’t mean it’s ready for production. True readiness requires operational support: monitoring, alerting, scaling, security, and performance.

Example:
You deploy a new service to prod. The logic works, but there’s no alerting on CPU/memory spikes. It goes down at midnight, and no one knows until users start complaining next morning.

Checklist to consider:

• Is there an SLA/SLO defined?

• Are alerts configured for error rates, latency, memory, CPU?

• Is there a fallback or graceful degradation?

2. Structured logging is essential for debugging

Plain text logs like Something went wrong are useless. Structured logs (e.g., JSON with trace IDs, timestamps, severity levels) allow automated parsing and correlation.

Example:
During a spike in 500 errors, you can’t trace which request failed where. But if logs include fields like trace_id, user_id, service, error_code, you can group and search easily.

Good structured log:

{
  "timestamp": "2025-06-19T10:22:10Z",
  "level": "ERROR",
  "service": "payment-service",
  "trace_id": "abcd1234",
  "message": "Payment gateway timeout"
}

3. Every network call should have a timeout

If you call another service without a timeout, your thread could hang indefinitely, leading to thread pool exhaustion and cascading failures.

Scenario:
Service A calls Service B. B gets slow. A keeps waiting. All threads in A get stuck → A becomes unresponsive → everything collapses.

Fix
Set timeouts at:

• HTTP client (e.g., OkHttp, RestTemplate)

• DB queries (JDBC socket timeout)

• Redis or Kafka clients

4. Rate limiting is critical, even for internal services

Internal systems can be misused or buggy. Rate limiting guards against flood or misuse - intentional or not.

Scenario:
An internal batch job starts hammering the notification service with 10K requests/sec due to a bug. Notification service goes down. Now even real users can’t get OTPs.

Fix:
Apply per-client or per-service rate limits using:

• API Gateway (e.g., Kong, NGINX)

• Service mesh (e.g., Istio)

• App-level logic (bucket/token/leaky bucket)

5. High latency impacts user experience as much as downtime

A slow service feels broken to users. Latency can degrade engagement, increase bounce rate, or trigger retries/load spikes.

Scenario:
Login API is up but takes 5s to respond due to DB slowness. Users refresh, causing even more load → full outage.

Fix:

• Track 95th/99th percentile latency.

• Add caching or DB query optimization.

• Offload heavy work to background jobs.

6. Retries should be bounded and controlled

Automatic retries can worsen an outage if uncontrolled - this is called a retry storm.

Scenario:
Service A retries B on failure every 100ms. B is already slow. Now it’s even more overloaded and goes into a full crash.

Fix:

• Use exponential backoff and jitter.

• Cap retry attempts (e.g., max 3).

• Add circuit breakers (e.g., Resilience4j, Hystrix).

7. Observability is more than just logs

You need logs, metrics, and traces (the “three pillars”) for full observability.

Scenario:
A request is slow. Without traces, you don’t know if DB, Redis, or another service is the bottleneck. Logs alone can’t answer this.

Fix:

• Logs → for deep inspection

• Metrics → for monitoring trends (e.g., Prometheus)

• Traces → for request flow (e.g., OpenTelemetry, Jaeger)

8. Feature flags must be maintained and cleaned up

Feature flags help with controlled rollout, but forgotten flags add tech debt and risk.

Scenario:
A bug is fixed but old flags still exist in code. Devs misunderstand logic later → reintroduce same bug.

Fix:

• Document flags (who owns them, expiry date)

• Use tooling like LaunchDarkly or Unleash

• Add flag cleanup checklist post-release

9. Every system has a performance bottleneck

No matter how optimized your code is, some component (DB, network, cache, CPU, etc.) will limit the system’s throughput or latency.

Scenario:
You scale your API horizontally but still hit latency issues. Root cause? A slow SQL join that’s CPU-bound and unindexed.

Fixes:

• Use tools like flame graphs, APM (e.g., New Relic), or p99 profiling.

• Continuously measure throughput vs. response time under load.

• Don’t guess bottlenecks - measure first.

10. Safe deployments require rollback mechanisms

If a release breaks production, you need a way to undo quickly, ideally with zero downtime.

Scenario:
A new release introduces a subtle bug in pricing logic. No one notices until revenue drops. But rollback is slow due to DB schema changes.

Best Practices:

• Use blue/green or canary deployments.

• Avoid irreversible migrations or release them behind a feature flag.

• Keep one-click rollback in your CI/CD pipeline.

11. Real-world failures differ from test failures

Staging ≠ prod. In production, you face network partitions, DNS failures, dirty data, and zombie jobs that tests never covered.

Scenario:
A service worked fine in staging. But in production, a downstream call times out on certain ISPs due to MTU mismatch - something tests never simulated.

Takeaways:

• Simulate chaos (e.g., Chaos Monkey).

• Validate under production-like load and data.

• Collect failure patterns from prod incidents.

12. API versioning is necessary for evolution

As APIs evolve, old clients must still work. Breaking changes without versioning = breaking everyone at once.

Scenario:
You change the payload format or rename a field. Suddenly, old mobile app clients start crashing.

Solutions:

• Use versioned URLs (/v1/user, /v2/user) or headers.

• Keep multiple versions live and sunset them gradually.

• Version your OpenAPI spec to stay in sync.

13. Avoid exposing internal error messages to users

Detailed stack traces or DB errors give attackers insight and confuse users.

Bad Example:

500 Internal Server Error: org.hibernate.JDBCException: could not execute statement

Fix:

• Show users friendly messages: “Oops! Something went wrong.”

• Log the internal error for engineers with request/trace ID.

• Apply exception mapping at controller boundaries.

14. Databases often become the scalability bottleneck

Most backends are read/write bound to a DB. Poor indexing, large joins, and lack of sharding are killers at scale.

Scenario:
As traffic grows, your Postgres query latency spikes. You realize a table with 50M rows isn’t partitioned and has no composite index.

Fixes:

• Design tables with future volume in mind.

• Regularly monitor slow queries.

• Add read replicas, caching layers, or move to scalable DBs (e.g., DynamoDB, CockroachDB).

15. Scheduled tasks must be monitored

Cron jobs or background workers often get no attention - until they stop running silently.

Scenario:
A cleanup job stops due to a bug in one row’s data. No alert. After a month, your DB is bloated, app is slow.

Fix:

• Log success/failure of every run.

• Set up alerts if job doesn’t run or fails X times.

• Use dashboards for job duration trends.

16. Rolling back code doesn’t undo external effects

Deploying an old version doesn’t rollback side effects like:

• DB rows written

• Emails sent

• Payments processed

Scenario:
A broken release sends duplicate invoices to 10K users. You roll back, but those emails are already in their inbox.

Best Practices:

• Use compensating transactions (e.g., “void invoice”).

• Track side effects with IDs (e.g., for retry safety).

• Keep side-effect actions behind flags or queues so they can be paused.

17. Caching systems and CDNs can become points of failure

We think of caches and CDNs as performance boosters - but they can break things just as easily. Stale data, cache stampedes, CDN outages - all can silently impact users.

Scenarios:

• A product price was updated, but due to CDN cache delay, users still see old pricing → trust issue.

• Redis goes down → app crashes because no fallback or TTL was handled.

Mitigation:

• Always set proper TTLs and cache headers.

• Implement cache fallback logic (e.g., fetch from DB if Redis fails).

• Invalidate cache explicitly after writes.

• Monitor cache hit ratios and propagation delays.

18. System behavior can change unexpectedly due to external dependencies

Your system is only as stable as what it depends on. Third-party APIs, DNS providers, TLS certs, cloud infra updates - any can break your flow without you changing a single line of code.

Scenario:
A payment provider silently changes their API response format → your parser crashes. Or their cert expires → your SSL handshake fails → transactions drop.

Precautions:

• Implement fallback logic and error categorization for third-party calls.

• Monitor dependency latency/availability as first-class metrics.

• Set alerts for cert expiration, DNS resolution, etc.

• Use contract testing or canary calls to detect API changes early.

19. High availability requires planning for failure modes

Explanation:
Redundancy isn’t optional. Systems will fail - the question is how gracefully you handle it.

Scenario:
Your DB goes down. No failover exists. Users hit a 500 wall for hours.

Design for HA:

• Replicate data across zones/regions.

• Add load balancer + health checks.

• Regularly test failover drills and chaos scenarios.

20. Cumulative latency across services adds up

Even if each service is “fast” (say 50ms), calling 5 such services adds 250ms. This hurts UX.

Scenario:
Your checkout page calls pricing, inventory, offers, and tax services - each <100ms. Combined latency = sluggish experience.

Fix:

• Run calls in parallel wherever possible.

• Use local caches for static data.

• Measure end-to-end latency, not just per-service metrics.

These aren’t just best practices - they’re survival strategies.
The sooner you internalize them, the fewer post-mortems you’ll write.
Whether you’re building APIs, backend systems, or distributed apps, treat this list as a checklist for production maturity.

Because writing code is easy. Running it at scale, safely? That’s where real engineering begins.

Share