Delivery Guarantees: At-Least-Once, Exactly-Once, and Idempotency
How message delivery guarantees work from the ground up: at-most-once (may be lost), at-least-once (may be duplicated), exactly-once (the goal), how idempotency turns a duplicate-prone system into an effectively-once one, why ordering matters, and what SQS, Kafka, and idempotency keys actually do.
A from-the-ground-up look at delivery guarantees, the last piece of the messaging arc. The three promises a system can make: at-most-once (a message may be lost but never duplicated), at-least-once (never lost but may arrive more than once), and exactly-once (once and only once). Why real systems run at-least-once and reach exactly-once in practice by making each operation idempotent, so a retried request has no extra effect, the way a Stripe idempotency key prevents a double charge. Why ordering matters, and what Amazon SQS standard and FIFO queues, Apache Kafka, and application-level idempotency keys actually guarantee. Everyday analogies, a hand-drawn diagram, and a visualizer per idea.
The previous piece was about the log: an append-only record that keeps every event and lets any reader replay it by moving its offset. Replay is useful, but it surfaces a hard question that has been under every piece in this arc. If a message can be retried, or replayed, or a worker can crash halfway through handling one, how do you make sure a customer is charged once and not twice? This closing piece is about the promises a messaging system can make, and how you keep them.
Imagine you have to mail a payment instruction to your bank: "move 100 dollars." A few things can go wrong in the post. The letter can get lost and never arrive. Or you can get nervous, mail a second copy to be safe, and now the bank receives two identical instructions and moves 200 dollars. Getting it to arrive exactly once, no more and no less, turns out to be the whole art. Messaging systems face the same three outcomes, and they have names.
At-most-once: send it and hope
Think of dropping a postcard in a mailbox and walking away. You never check whether it arrived. Most of the time it does; once in a while it is lost, and you never find out. What you can be sure of is that the person never gets two copies, because you only ever sent one.
That is at-most-once delivery. The sender fires the message and does not wait for confirmation or retry. In the words of the Kafka design docs, "messages are delivered once, and if there is a system failure, messages may be lost and are not redelivered." You trade safety for speed and simplicity: no acknowledgment to wait for, no bookkeeping, and never a duplicate, but a message can silently vanish.
Worked example. A "user is typing" indicator is sent at-most-once. If one update is lost, the little dots flicker for a moment and nobody cares, so paying for retries and duplicate-protection would be wasted effort. This is exactly the Redis pub/sub behavior from the pub/sub piece: fire-and-forget, no persistence, gone if you missed it.
The sender fires one message and does not wait for any reply.
At-least-once: keep trying until you hear back
Now picture a tracked letter with a return receipt. You mail it, and you wait for the little card that says "delivered." If the card does not come back in time, you assume the letter was lost and mail it again. Your letter will definitely arrive, that is the point, but there is a catch: if the letter actually arrived and it was the receipt that got lost, you will mail a second copy and the recipient now has two.
That is at-least-once delivery, and it is what most real systems use. The Kafka docs put it plainly: "messages are delivered one or more times. If there is a system failure, messages are never lost, but they may be delivered more than once." The sender keeps retrying until it gets an acknowledgment, so nothing is ever lost. The price is duplicates: a message whose acknowledgment was lost gets sent again, and the receiver sees it twice. At-least-once is the default guarantee in Kafka, and it is what an SQS standard queue gives you from the message-queues piece.
Worked example. A worker pulls "charge card 100 dollars," charges it, and then crashes before it can send the acknowledgment. The queue never heard "done," so after the visibility timeout it hands the same message to another worker, which charges the card a second time. Nothing was lost, but the customer just paid twice. The duplicate is not a bug in the queue; it is the guarantee working as designed. The fix is in the next section.
The sender sends the message and waits for an acknowledgment.
Exactly-once, in practice: make the work idempotent
Exactly-once is the third promise, and the one everyone wants: "each message is delivered once and only once. Messages are never lost or read twice even if some part of the system fails." The honest truth is that across an unreliable network you cannot stop a message from being sent twice; what you can do is make the second copy have no extra effect. Do that, and the outcome is exactly-once even though delivery was at-least-once. This is the practical route almost everyone takes, and it rests on one idea: idempotency.
Here is the everyday version. An elevator call button is idempotent: pressing it once calls the elevator, and pressing it five more times changes nothing, because the elevator is already coming. A light switch flipped to "on" is idempotent: flip it to on again and it is still just on, not "extra on." An operation is idempotent when doing it a second time produces the same result as doing it once. If your message handler is idempotent, a duplicate delivery is harmless: the redundant copy lands, the handler notices it already did the work, and nothing happens twice.
The standard way to make an operation idempotent is an idempotency key: the sender attaches a unique id to the request, and the receiver remembers which ids it has already processed, so a repeat is recognized and skipped. This is exactly how Stripe's payment API prevents double charges. In Stripe's own words, its idempotency "works by saving the resulting status code and body of the first request made for any given idempotency key," and "subsequent requests with the same key return the same result." The client generates the key as a random value ("we suggest using V4 UUIDs"), and Stripe keeps each key for at least 24 hours.
Worked example. The worker from the last section attaches idempotency key a1b2 to the charge. The first
time, the card is charged and a1b2 is recorded as done. When the crash causes a retry, the same message
arrives with the same key a1b2; the handler sees it has already processed a1b2, returns the saved result,
and does not touch the card again. Delivery happened twice; the charge happened once. That is exactly-once in
practice. (Inside its own boundary, Kafka can also give you this automatically: its idempotent producer, from
version 0.11, guarantees "resending a message will not result in duplicate entries in the log," and
transactions extend that across a read-process-write pipeline.)
The sender attaches a unique idempotency key, a1b2, to the charge.
Keep it in order
There is a second promise that often matters as much as not-losing and not-duplicating: keeping messages in the order they were sent. Picture a single-file line at a bank counter. As long as everyone is served in the order they arrived, the story stays consistent. Let people cut ahead and the outcome can change.
Ordering means a consumer sees messages in the same sequence a producer sent them. It matters whenever the effect of one message depends on an earlier one. From the log piece, recall that ordering is guaranteed only within a single partition, and that routing by a key (a customer id) keeps that customer's events on one partition and therefore in order. A FIFO (first-in, first-out) queue makes the same promise for a queue.
Worked example. Two events for one account: "deposit 100," then "withdraw 50," on an account that starts at zero and may not go negative. In order, the balance goes 0, then 100, then 50: fine. Reordered to "withdraw 50" first, the withdrawal hits a zero balance and is rejected, and you end at 100 instead of 50: a different, wrong answer, from the very same two messages. Keeping both events on the same partition (or in the same FIFO group) is what prevents it. Note the tension with the log piece: splitting across many partitions buys speed but only preserves order within each one, so you key related events together to get both.
Two events for one account that starts at 0 and may not go negative: deposit 100, then withdraw 50.
What actually runs
The guarantee you get is a property of the tool and how you configure it.
Google Cloud Pub/Sub is at-least-once by default. Exactly-once delivery can be turned on for a pull subscription, with the guarantee that "no redelivery occurs after the message is successfully acknowledged," and ordering is opt-in through an ordering key, where messages that share a key "are expected to be received in order." Even Google Cloud Tasks, the managed queue, is at-least-once and warns you outright to "ensure that there are no harmful side-effects of repeated execution," which is just this section restated.
Amazon SQS offers both. A standard queue is at-least-once with best-effort ordering, so it is fast and can deliver a message more than once and occasionally out of order. A FIFO queue preserves order within a message group and does the deduplication for you, giving exactly-once processing at a lower throughput.
Apache Kafka is at-least-once by default; its idempotent producer removes producer-retry duplicates in the log, and transactions give exactly-once semantics across a consume-process-produce pipeline within Kafka.
Application-level idempotency keys, the Stripe pattern, are the portable answer that works across any transport: make the handler idempotent and duplicates stop mattering, whatever queue or log delivered them.
The rule of thumb: assume at-least-once, because that is what robust systems actually deliver, and make your handlers idempotent so a duplicate is harmless. Reach for FIFO or Kafka transactions when you also need strict order or built-in dedup, and accept the lower throughput that buys.
That closes the messaging arc: a queue to buffer work and spread it across workers, pub/sub to fan one event out to many services, the log to keep and replay a durable history, and delivery guarantees to make it all correct under retries. Together they are the backbone that lets independent services talk without waiting on each other. The system-design series continues from here, moving up from these building blocks to how full systems are put together.
Sources: Message delivery guarantees for Apache Kafka (Confluent); Idempotent requests (Stripe API); Exactly-once delivery (Google Cloud Pub/Sub); Message ordering (Google Cloud Pub/Sub).
