Idempotency keys: stopping duplicate webhook processing

Duplicate delivery is worse because it can produce a perfectly plausible system state with an economically incorrect outcome: two charges, two shipments, two invoices, two provisioning jobs, or two welcome emails sent from an automation that insists it ran only once.

This is not an exceptional failure mode. Webhook providers generally deliver events under an at-least-once model. If a network timeout, proxy failure, connection reset, or slow consumer prevents a successful response, the provider retries. Your application therefore has to assume that the same logical event may arrive repeatedly, sometimes while the first attempt is still running.

Idempotency keys in webhooks for duplicate prevention are the boundary between a resilient custom web application and a brittle chain of triggers that merely works when the network behaves. The key itself is simple. The architecture around it is not.

The reality of at-least-once delivery: why webhooks repeat

A webhook is not a function call with a guaranteed return path. It is a message sent across infrastructure that neither side fully controls. The provider sends an HTTP request; your endpoint receives it, performs some work, and returns a response. At every point in that sequence, the provider may lose certainty about what happened.

Suppose your endpoint receives a payment-success event and immediately creates an invoice. The invoice is committed to your database. A moment later, the process stalls before returning the provider’s expected success response. From the provider’s perspective, delivery failed. It retries. Your endpoint receives the same event and creates a second invoice.

Nothing in the transport layer knows that the first invoice already exists. The provider is not necessarily misbehaving. Retrying is the rational response to uncertainty.

The reverse is also possible. Your application may return a response quickly, but an internal queue, database connection, or downstream API may fail after the response has been sent. The provider records a successful delivery while your business operation remains incomplete. Idempotency does not solve every reliability problem, but it gives the processing layer a stable identity with which to coordinate retries and recovery.

The timing constraints make the problem sharper. Shopify enforces a five-second response deadline before retry behavior can be triggered, while GitHub expects a response within ten seconds. Other providers use different thresholds and retry schedules. Treating webhook receipt as a place to perform every business operation is therefore a structural mistake: the endpoint should authenticate, identify, record, and dispatch, not conduct a long-running workflow while the sender waits.

At-least-once delivery is not a provider defect to be worked around. It is the contract your webhook consumer must be designed to survive.

The duplicate is not always byte-for-byte identical

A provider may resend the same event with an identical payload, but an implementation should not build its entire identity strategy around raw equality. Headers may differ. Serialization details may change. A proxy may preserve the body while altering transport metadata. Some systems offer a stable event identifier; others give you only request content and provider-specific context.

The safest identity source is usually a provider-issued event ID, scoped to the provider and the environment in which it was received. An event ID from a test account must not collide with an event ID from production. If the provider does not expose a trustworthy identifier, the application may derive a key from the exact raw request bytes and relevant source metadata.

That distinction matters because parsing and re-serializing JSON destroys information that can affect a hash. JSON objects can contain the same logical data while differing in key order or whitespace. Hashing a parsed object after your framework has normalized it may make two distinct requests look the same—or make equivalent retries look different, depending on the serialization path.

Anatomy of an idempotency key

An idempotency key is not merely a random string attached to a request. It is a durable claim that several deliveries refer to the same logical operation.

The key should answer a narrow question: has this particular operation already been accepted or completed within the relevant scope?

That scope is part of the design. A useful key often includes, conceptually, four dimensions:

• The event or operation identifier supplied by the provider.
• The provider or source system that owns the identifier.
• The environment, such as production or staging.
• The business operation being protected, when one event can legitimately drive more than one operation.

The last dimension is easy to overlook. An order-created event might lead to fulfillment, invoice generation, analytics ingestion, and customer notification. Those operations may have different retry semantics and different notions of completion. One global “processed” flag can therefore become dangerously coarse. Marking the event as complete after analytics ingestion does not mean fulfillment succeeded.

A practical idempotency record commonly needs more than a Boolean. It may contain:

The record is not the business operation itself. It is the coordination layer around that operation. It tells concurrent workers which attempt owns execution, which attempts must wait or stop, and whether a previous result can be returned safely.

Keys must be stable, bounded, and scoped

Provider limits are not decorative documentation. They are constraints that belong in the application’s data model and validation layer.

Stripe limits idempotency keys to 255 characters and retains them for 24 hours. Square Payments API limits its idempotency key input to 45 characters, while Square Orders allows up to 192 characters. A key-generation strategy that emits a verbose JSON blob may work against one integration and fail at the next boundary.

A compact construction is usually superior: a provider namespace, environment marker, operation label, and stable event identifier, separated or encoded predictably. If the provider’s identifier is already unique within the appropriate scope, do not inflate the key with redundant payload material. If a digest is needed, use a fixed-length representation rather than an unbounded serialization.

The application should also reject a dangerous condition: the same key arriving with materially different content. That is not a harmless retry. It can indicate a provider bug, a collision, a consumer-side key-generation defect, or an attempted replay with altered data. The correct response is to quarantine the inconsistency and alert, not silently process whichever payload happened to arrive last.

Atomic locking strategies: Redis and database primitives

The central race condition looks like this:

1. Worker A checks whether the idempotency key exists.

2. Worker B checks whether the same key exists.

3. Both observe that it does not.

4. Both begin the side effect.

A separate “check” followed by a separate “insert” is not protection. It is a timing window wearing the costume of protection.

The claim must be atomic. Redis provides a direct primitive through SET with NX and EX options: create the key only if it does not already exist, and attach an expiration at the same time. The combined operation prevents two workers from both believing they acquired the right to process the event.

A successful claim can store a short-lived processing marker. A failed claim means another worker has already claimed the key. The second worker must then inspect the existing state rather than proceed optimistically.

A conceptual processing sequence is:

1. Authenticate the request and capture the raw body.

2. Derive a stable, scoped idempotency key.

3. Atomically claim the key with a bounded lease.

4. If the claim fails, inspect the existing record.

5. If the operation is completed, return the recorded result or acknowledge the duplicate.

6. If the operation is still processing, avoid concurrent side effects and apply a deliberate retry or polling policy.

7. If the lease is expired, recover according to the operation’s state and observability data.

8. Execute the business operation.

9. Persist completion and the result reference.

10. Acknowledge the webhook.

The phrase “return the recorded result” matters. A duplicate request should not merely receive a generic success response if downstream systems depend on a specific object identifier, status, or provider-facing response. Storing a result reference allows the duplicate path to be deterministic.

Redis is fast, but speed is not the same as durability

Redis is attractive because its atomic commands are concise and its latency is low. It is often an excellent first gate for high-volume webhook ingestion. But a Redis lock should not be treated as an immutable historical record unless the deployment and persistence guarantees justify that decision.

The business database remains the authoritative place for durable operation state in many systems. A robust design may use Redis to suppress concurrent duplicate execution while storing the operation record, unique key, status, and resulting business identifiers in a transactional database.

This produces a layered defense:

• Redis prevents a hot duplicate from entering the critical section twice at the same instant.
• A database uniqueness constraint prevents durable duplicate records even if workers, processes, or infrastructure bypass the cache.
• Business-level constraints prevent the side effect from being repeated when the integration reaches an external system.

The database constraint is particularly important. A unique index over the scoped idempotency key turns a race into a controlled conflict rather than a duplicate row. The insert and the initial processing state should be created transactionally where possible. If the database says the key already exists, the worker must load that record and follow its state rather than attempting a second operation.

Locks do not make external APIs magically idempotent

A local claim protects your application from concurrent execution. It does not guarantee that an external payment, shipping, or provisioning API will receive only one request.

Consider the failure sequence in which your worker acquires the key, calls a downstream API, and then crashes before recording the downstream response. When the webhook is retried, your application may see an incomplete local record. If it simply calls the downstream API again, the duplicate has moved beyond the webhook boundary.

For any external side effect that supports idempotency, propagate a stable operation key to that API as well. The key should represent the downstream operation, not necessarily the entire incoming webhook. One order event might produce separate idempotent keys for payment capture, fulfillment creation, and invoice issuance.

Where the downstream service offers no idempotency mechanism, the application needs a reconciliation strategy: search by a stable merchant reference, query the provider before creating a new object, or use a durable outbox and manual review path for ambiguous outcomes. Retrying blindly after an unknown result is how double processing becomes a financial incident.

Hashing pitfalls: raw bytes versus serialized JSON

Payload hashing is often proposed as the universal fallback: parse the JSON, serialize it in a normalized form, hash the result, and use the digest as the idempotency key. It sounds elegant. In practice, careless normalization produces a brittle identity function.

Hashing re-serialized JSON bodies directly can fail because key ordering and whitespace may change. Two retries with equivalent content can generate different serialized representations if different libraries, middleware layers, or canonicalization rules are involved. Conversely, a naïve normalization process may erase distinctions that are meaningful to the provider or to your business logic.

The raw request body should be captured before JSON parsing whenever signature verification or content hashing depends on it. Exact raw bytes preserve the payload as transmitted. If the provider supplies a stable event ID, that is generally preferable because it expresses the provider’s own identity model rather than forcing your application to infer one.

A sound fallback hierarchy is:

1. Use a verified provider event ID, scoped by source and environment.

2. If no event ID exists, hash the exact raw request bytes together with stable source context.

3. If the provider documents a canonicalization scheme, follow that scheme precisely.

4. Never assume that a framework’s parsed object and subsequent JSON serialization preserve the original identity.

There is another subtlety: a content hash identifies a payload, not necessarily an operation. The same payload might legitimately be submitted twice as two separate operations, while two payloads with minor metadata differences might represent the same business action. Hashing is therefore a fallback mechanism, not a substitute for understanding the provider’s event semantics.

Signature verification belongs before business interpretation

The raw body is also essential for webhook signature verification in systems that sign the transmitted payload. Parsing first and reconstructing later can invalidate the signature or make verification dependent on framework behavior.

The secure intake path should verify authenticity, identify the source, and establish the idempotency key before any business-side interpretation. Do not write a “processed” record for an unauthenticated request simply because its JSON contains a familiar event name. Otherwise, an attacker can poison the deduplication store or reserve keys that legitimate deliveries need.

Authentication and idempotency solve different problems:

• Signature verification answers whether the request plausibly came from the expected provider.
• Idempotency answers whether this authenticated operation has already been claimed or completed.

Confusing the two leads to either weak security or unsafe retry handling.

TTL policies and state management

Every idempotency record has a retention question attached to it. How long should the application remember that a key was processed?

A 24-hour period is a common baseline in API idempotency systems, and Stripe retains its idempotency keys for 24 hours. But a generic TTL is not a universal law. The correct window depends on the provider’s retry behavior, the business cost of duplication, the expected delay in manual replay, and the period during which an operation can be reconstructed safely.

A short TTL reduces storage, but it can reopen the duplicate window. If a provider retries after the record expires, the application may process the event again. A long TTL preserves more history, but it introduces storage and data-retention concerns and can make legitimate replays harder to distinguish from accidental duplicates.

The right answer is usually to separate coordination leases from deduplication history.

A processing lease answers: how long may this worker be considered the owner before another worker can recover the operation?

A completed-operation retention period answers: how long should the system remember that the side effect already happened?

Those periods should not be the same. A worker may need a lease measured in minutes, while a completed payment or provisioning operation may need a durable record far beyond the provider’s ordinary retry window.

The state machine is more important than the lock

A lock alone says “someone is working.” It does not say whether the work succeeded, failed before the side effect, failed after the side effect, or is waiting for an external dependency.

A useful state model distinguishes at least:

• processing: a worker has claimed the operation.
• completed: the intended side effect has been confirmed and its result recorded.
• failed_before_effect: the operation can be retried safely.
• unknown_outcome: the worker may have reached an external system, but confirmation was lost.
• dead_lettered: automated retry is no longer appropriate without inspection.

The unknown_outcome state is where shallow implementations collapse. If the worker times out while calling a payment or shipping API, the result is not automatically “failed.” It may have succeeded remotely. Retrying without reconciliation can create the duplicate you designed the idempotency layer to prevent.

This is why a durable operation record should contain external references, attempt timestamps, and error context. A monitoring dashboard that shows only HTTP status codes will not tell you whether a business side effect is safe to repeat.

A no-code workflow still needs this architecture

Visual automation platforms often make webhook handling look deceptively linear: trigger, create record, send message, update status. The interface hides the concurrency model, but it does not remove it. If the trigger platform retries, if two runs start in parallel, or if a downstream connector times out after creating a record, the same duplicate-processing problem exists beneath the canvas.

For no-code and low-code systems, idempotency can be implemented through:

• A dedicated operation table with a unique key column.
• An atomic “create if absent” action, if the platform genuinely guarantees it.
• A database-level unique constraint rather than a visual existence check.
• A status field that distinguishes active processing from completed work.
• A delayed reconciliation workflow for ambiguous external results.
• Provider-side idempotency keys passed through connector configuration where supported.

The dangerous pattern is a visual sequence that first searches for a record and then creates one if no record appears. Two workflow runs can perform the search before either creates the record. That is the same race condition as in custom code, merely rendered in larger buttons.

No-code does not replace traditional engineering here. It changes where the engineering constraints are expressed. The durable design still requires transaction boundaries, unique constraints, retry semantics, and an explicit answer to the question of what happens after a timeout.

Designing the webhook boundary for fast acknowledgement

A webhook endpoint should be deliberately boring. Its job is to validate the request, establish identity, record the event, and hand off work. It should not become a bloated orchestration engine because the first version happened to have only one side effect.

A resilient boundary usually follows this shape:

1. Receive the request and preserve the raw body.

2. Verify the webhook signature and source.

3. Extract or derive the scoped idempotency key.

4. Atomically create an intake or operation record.

5. Return an acknowledgement within the provider’s response window.

6. Process the operation asynchronously.

7. Update the operation state and persist result references.

8. Reconcile failures and unknown outcomes separately from ordinary retries.

This approach is especially important under strict provider deadlines such as Shopify’s five seconds and GitHub’s ten seconds. Returning quickly does not mean falsely claiming that the business operation is complete. It means acknowledging receipt while moving the actual work into a controlled processing path, according to the provider’s accepted response semantics.

The asynchronous worker must still be idempotent. Queuing an event does not eliminate duplicates; it can multiply them if every retried webhook creates another queue message. The queue consumer needs the same atomic claim and state checks as the synchronous endpoint.

Observability must follow the idempotency key

Logs should make one operation traceable across the entire stack. Include the scoped idempotency key, provider event ID, internal operation ID, downstream request key, attempt number, and final state wherever those values can be recorded safely.

Metrics should distinguish:

• First-time claims from duplicate deliveries.
• Duplicates that arrived during active processing.
• Duplicates resolved from completed results.
• Expired leases recovered by another worker.
• Unknown external outcomes.
• Key-content conflicts.
• Downstream idempotency failures.

A rising duplicate count does not necessarily indicate an application defect; providers may retry because of network conditions or response timing. A rising count of completed operations being re-executed does indicate a defect in the consumer. Without these distinctions, the dashboard reduces every failure to “webhook received” and leaves the engineering team guessing.

Common implementation mistakes

The same flawed approaches recur across custom web applications, SaaS backends, and automation workflows.

1. Checking before inserting.

A read followed by a write is not atomic. Concurrent workers can both pass the check. Use a unique constraint or an atomic claim primitive.

2. Using a random key per delivery.

A new UUID generated every time the webhook arrives identifies attempts, not the logical operation. Retries will never collide and duplicates will pass through.

3. Treating a processing marker as completion.

A crashed worker can leave the marker behind. The system needs leases, timestamps, and recovery rules rather than an eternal Boolean.

4. Deleting the key after success.

This removes the only durable evidence that the operation already happened. A later retry becomes indistinguishable from a new event.

5. Hashing parsed JSON without a canonicalization contract.

Middleware may reorder fields, remove insignificant whitespace, or normalize values. Use raw bytes or a provider-issued event ID instead.

6. Returning a success response only after all side effects finish.

Slow processing increases provider retries and widens the race window. Acknowledge receipt promptly and process asynchronously where the provider allows it.

7. Assuming the local database protects an external side effect.

Your unique invoice row does not prevent a payment API from charging twice. Propagate idempotency downstream or reconcile ambiguous results.

8. Using one event-level flag for unrelated operations.

Marking an event as processed after one branch succeeds can suppress another branch that still needs execution. Model operation identity at the level where duplication would cause harm.

The strict engineering position

Idempotency keys are not a decorative header and not a cache optimization. They are part of the correctness model for any system that consumes retried messages and performs side effects.

The minimum respectable implementation has a stable, scoped identity; an atomic claim; a durable state record; a distinction between active, completed, failed, and unknown outcomes; a retention policy that matches the business risk; and downstream protection wherever another API can create money, inventory, access, or legal records. It preserves raw request bytes when identity or signature verification depends on them. It treats database uniqueness as a final line of defense rather than trusting a cache alone.

The mandate is strict because the failure is strict: never process an externally triggered side effect on the assumption that the request is unique. Assume retries. Assume concurrency. Assume timeouts can occur after the remote system has succeeded. Then make the operation’s identity durable enough that the next worker can determine what happened without guessing.

That is the difference between a webhook integration that survives ordinary network behavior and one that merely survives a demo.