Idempotency

Recall the irreducible ambiguity from the overview: when a request times out, you cannot tell whether it ran. The only sane response is to retry — but a retry of a request that did run creates a duplicate, and a duplicate “charge the customer” or “ship the order” is a catastrophe. Idempotency is the discipline that breaks this trap: design operations so that doing them twice has the same effect as doing them once. Then duplicates stop being dangerous, and retries become free. This page is about why the network forces this on you, and how to build it.

The forcing function: the network delivers duplicates

It’s worth saying plainly, because it reframes idempotency from a nice-to-have into a structural requirement: you do not get to choose whether duplicates happen. They happen because:

Client ──"charge $50"──► Server   [server charges, sends OK]
                          │
   reply lost in network ◄┘
Client times out, doesn't know it worked, RETRIES
Client ──"charge $50"──► Server   [server charges AGAIN — $100!]

The lost reply is indistinguishable from a lost request (see Timeouts, Retries & Backoff). Any reliable client must retry, so any robust server must tolerate the duplicate. The duplicate is not a bug to prevent; it’s a condition to absorb.

What “idempotent” actually means

An operation is idempotent if applying it multiple times yields the same result as applying it once. The word comes from math (a function f where f(f(x)) = f(x)), but the systems definition is about observable effect on state, not return value.

Some operations are naturally idempotent; some are not:

Operation	Idempotent?	Why
SET balance = 100	Yes	Re-running lands on the same state
balance = balance + 50	No	Each run adds again
HTTP GET, PUT, DELETE	Yes (by spec)	Same target state regardless of count
HTTP POST	No (by spec)	“Create a new resource” each time
DELETE WHERE id = 7	Yes	After the first, it’s already gone

Idempotent ≠ safe ≠ same response

A DELETE is idempotent (end state is identical) but not “safe” (it changes state). And idempotent doesn’t mean every call returns the same response — the second DELETE might return 404 instead of 200. What’s invariant is the resulting state of the system, which is exactly what duplicates threaten.

Idempotency keys: making anything idempotent

The non-idempotent operations are the dangerous ones (POST /charges, “place order”), and you can’t always rewrite them into naturally-idempotent form. The general solution is the idempotency key: a unique token the client generates for a logical operation and attaches to every retry of it. The server uses the key to recognize and collapse duplicates.

Client generates key once:  K = "a1b2-c3d4"  (e.g. a UUID)

Attempt 1:  POST /charges  Idempotency-Key: a1b2-c3d4  →  (lost reply)
Attempt 2:  POST /charges  Idempotency-Key: a1b2-c3d4  →  server: "I've
                                                            seen K, here's
                                                            the SAME result"

   The customer is charged exactly once, no matter how many retries.

The server-side logic, conceptually:

on request with key K:
  look up K in a dedup store
  ├─ not seen:  process it, atomically store (K → result), return result
  ├─ seen, done: return the STORED result (don't re-run the side effect)
  └─ seen, in-flight: reject or wait (a retry arrived before the first finished)

Two details make or break correctness. First, the key must be generated by the client, once per logical operation, and reused across retries — if the client makes a fresh key per attempt, the server sees them as different requests and the protection vanishes. Second, storing the result and performing the side effect must be atomic (or at least the side effect must itself be idempotent under the key), or a crash between the two reintroduces the duplicate. This is why payment APIs like Stripe make Idempotency-Key a first-class header.

Deduplication in message systems

The same pattern appears in messaging. Most queues guarantee at-least-once delivery — they’d rather deliver a message twice than lose it — so consumers receive duplicates routinely. The consumer defends itself by deduplicating on a message ID:

consumer receives message with id M:
  if M in processed-set:  skip (already handled)
  else:                   process M, then record M as processed

This is idempotency wearing different clothes: the message ID is the idempotency key. The closely related (and frequently misunderstood) goal of making the whole pipeline behave as if each message took effect once — even across failures — is exactly-once semantics, which is built from at-least-once delivery plus idempotent processing. We unpack why true exactly-once delivery is impossible but exactly-once effect is achievable in Exactly-Once Semantics.

Idempotency turns the hard problem into an easy one

You cannot make the network stop duplicating messages — that’s physics and timeouts. But you can make duplicates harmless, and once they’re harmless, aggressive retries (with backoff) become a feature instead of a hazard. The entire reliability stack rests on this swap: stop fighting duplicates, start absorbing them.

What it buys us, and what it costs

Idempotency buys you the freedom to retry without fear — the single most important property for building reliable systems on an unreliable network. It converts the network’s worst habit (delivering the same thing twice) into a non-event, and it’s what makes timeouts, retries, queues, and failover safe to use at all. The cost is real but modest: you must thread idempotency keys through your APIs, maintain a dedup store (with a retention window and the storage that implies), and reason carefully about the atomicity of “do the thing and record that you did it.” That bookkeeping is the price of admission to a distributed system that doesn’t double-charge its customers — and it is always cheaper than the alternative.

Check your understanding

1. Why does the ambiguity of a timed-out request force both retries and idempotency, rather than either being optional?
2. Classify these as idempotent or not, and say why: SET x = 5, x = x + 1, HTTP PUT, HTTP POST.
3. Why must an idempotency key be generated by the client once per logical operation and reused on retries, rather than freshly per attempt?
4. Why must “perform the side effect” and “record the idempotency key/result” be atomic? What goes wrong if a crash falls between them?
5. How is consumer-side deduplication in a message queue the same idea as an idempotency key, and how does it relate to exactly-once semantics?