Nobody calls an agency because their integration is down. They call because revenue is missing and nobody can say where it went. Half the rescue projects we take on trace back to the same root cause: two systems glued together with a webhook and a prayer, working perfectly in every demo, silently dropping data in production. This post is the anatomy of that failure and the deliberately boring architecture that prevents it.
How a webhook loses an order
The naive integration is beautifully simple. Shopify fires a webhook, your endpoint creates the order in the ERP, done. Its failure modes are just as simple, and each one is invisible:
- Your endpoint was deploying — or crashed, or timed out — when the webhook fired. The sender retries a few times, then gives up. Forever.
- The payload arrived, but a downstream API call inside your handler failed. You returned 200 anyway. The sender thinks you succeeded.
- The same event arrived twice, and now there are two orders and one very confused warehouse.
None of these throw an alert, because from each system's point of view nothing went wrong. The order simply stops existing in one of the two places. You find out when a customer calls, weeks later, and reconciliation becomes archaeology.
The boring architecture that survives
Every integration we ship has the same three-part shape, and none of it is novel — that is the point.
Receive, persist, acknowledge — then process
The webhook handler does exactly one thing: write the raw event to a durable inbox table and return 200. Processing happens in a separate worker reading from that inbox. If the ERP is down, the event waits; nothing is lost because acknowledgement and processing were never coupled.
Make every operation idempotent
Every event carries an ID, and every handler checks it before acting. A duplicate delivery becomes a no-op instead of a duplicate order. This one property converts "retries are dangerous" into "retries are free", and the entire system relaxes around it.
Reconcile on a schedule, not on suspicion
Webhooks are a notification channel, not a source of truth. A nightly job compares both systems and flags drift — because the only integrations that stay correct are the ones something is actively checking. The reconciler regularly catches the failures nothing else could: the webhook the sender never fired, the event that arrived malformed, the edge case nobody imagined.
A healthy integration is not one that never fails. It is one where failure is loud, bounded, and recoverable by retrying.
What to ask about the integrations you already have
If a vendor built your integrations, three questions reveal which architecture you got:
- If our endpoint is down for an hour, what happens to the events fired during it?
- If the same event is delivered twice, what exactly happens?
- What process, human or automated, would notice if the two systems disagreed?
"The webhook just handles it" as an answer to any of the three means you have the demo version. It will keep working right up until the moment it matters.
We audit inherited integrations as a fixed-scope engagement — get in touch if you would rather find the silent failures before your customers do.