Nobody notices when the orders stop

Most people picture a commerce outage as the site being down. A 503, a status page, somebody's pager going off. Those are bad, but they're honest. Everyone finds out within about ninety seconds, and the whole company knows what to do.

The incidents I remember are the quiet ones. The store is up. Checkout works. Customers are paying. Orders are landing in the database exactly like they should. And somewhere between the storefront and the ERP, nothing has moved for six hours, and the first person to notice is someone in the warehouse asking why the morning's pick list is empty.

That's the failure mode worth designing against, because every layer of your monitoring is pointed somewhere else.

Why it hides

Uptime monitoring watches whether the site responds. Application monitoring watches error rates and slow transactions. Both of those were green the whole time, because nothing threw an error. A queue consumer exited. Or the ERP started returning a shape the mapping layer didn't expect and the handler swallowed it. Or someone rotated a credential on the other side and the retry logic did exactly what it was told, which was retry forever, silently.

The store is a perfectly healthy system publishing messages into a void.

I spent a few years building middleware between commerce platforms and ERPs (NetSuite, SAP, ERPNext, a handful of others), and the pattern showed up on every single one of them, regardless of which systems were on either end. The technology varied. The failure shape didn't.

Don't let the storefront call the ERP

The first decision that matters is one you make before writing any integration code: the storefront should never call the ERP synchronously.

It's tempting, because it's simple, and because in staging the ERP always answers. Then it's Black Friday, the ERP is doing an inventory job, and every checkout is now holding a connection open waiting for a system that has no idea it's on the critical path. You've coupled your revenue to somebody else's batch window.

Publish an event instead. Let a consumer pick it up. Put a mapping layer in between so neither side has to know the other's data model. In Magento that's communication.xml defining the topic, queue_topology.xml for the exchange, queue_consumer.xml binding the handler, and a consumer process doing the work out of band. The storefront's job ends when the message is published.

Then there are four things you need, and you need all four, because each one covers a failure the others don't.

Idempotency

The same order will reach the ERP twice, and I mean will rather than might. A consumer dies after the ERP commits but before the ack, the message goes back on the queue, and it gets redelivered.

If the receiving side keys on your external order ID and no-ops when it's already seen it, this is a non-event. If it doesn't, you've created a duplicate sales order, and somebody in finance finds it three weeks later.

Idempotency is cheap to build on day one and genuinely painful to retrofit, because by then you have to figure out which of the duplicates were real.

Retries with somewhere to fail

The ERP goes down for maintenance. That's normal. Your integration should retry with backoff and carry on.

What matters is where a message goes when it has exhausted its retries. If the answer is "it disappears," you have built a system that loses orders silently, which is the thing we started out trying to avoid. A dead-letter queue is what gets you from "we lost some orders" to "we lost fourteen orders and here they are."

Reconciliation

Eventually the two sides drift. Not because anything dramatic broke, but because of a partial failure six weeks ago, or a manual edit somebody made in the ERP, or a record that was updated on one side during a deploy.

A scheduled job that compares both sides and reports the difference is unglamorous and it is the only thing that catches this class of problem. The alternative is that finance catches it, which is later and worse.

Delta over full sync

Full catalog syncs are how you take a store down at two in the morning. If you can sync only what changed, sync only what changed. Keep the full sync as the thing you run deliberately when you need to rebuild, not the thing that fires every night because it was easier to write.

Monitor the flow, not the box

Here's the part that actually closes the loop. All four of those things can be implemented correctly and you can still sit in the failure I opened with, because none of them tell you the pipe has stopped.

Consumer lag needs to be a first-class metric, monitored and alerted on exactly like uptime is. Checking whether the consumer process is running will fool you, because a consumer that is quietly failing every message looks identical from the outside to one doing its job. Queue depth over time. Age of the oldest unacknowledged message. Orders created in the last hour versus orders acknowledged downstream in the last hour, and an alert when those two numbers diverge.

That last one is almost embarrassingly simple, and it's the single most useful alert I've ever set up for a commerce platform. Two counts and a threshold. It catches the entire category.

The bit I keep coming back to

Commerce integration work gets described as plumbing, and there's some truth to that. Nobody writes a case study about a reconciliation job.

The interesting engineering lives outside the happy path. An order placed, an order arriving downstream, anyone can build that in an afternoon. The work is everything that happens when one side is slow, or lying, or has been restarted mid-transaction, and whether your system notices out loud or fails politely and quietly.

Systems that fail loudly are easy to run. Systems that fail politely will cost you a weekend.