There's a particular thread that resurfaces on r/shopify every few weeks, almost word for word. A merchant gets an order, opens it up to fulfil, and sees the little orange diamond: high risk of fraud. They click in for an explanation and get one sentence, "this order has characteristics similar to fraudulent orders observed in the past", and nothing about which characteristics. So they do what the thread always asks: "It's a $180 order, do I ship it or not?" And the replies are a coin-toss of strangers guessing on a stranger's behalf.

That is the strange thing about Shopify's fraud flag. It is the most-used piece of automation in the whole platform, it runs on every single order, and it's also the one merchants resent most, because it doesn't actually decide anything. It scores the order, raises an eyebrow, and hands the judgement call straight back to a human who has even less information than the algorithm that flagged it.

A good automation removes a decision. The high-risk flag just relocates it, from the machine that has the data to the person who doesn't.

Two ways to be wrong, and both are expensive

Every flagged order is a fork with a bad branch on each side. Fulfil it and you might be shipping product to someone using a stolen card, and when the real cardholder notices, you eat a chargeback. On Shopify Payments, that amount is simply pulled from your next payout, you're usually out the merchandise and the shipping, and the dispute counts against you whether you win it or not. The average chargeback now runs about $191, up from $165 a few years ago, and industry figures put the all-in cost of fraud at roughly $3.75 lost for every $1 of the original order.

So merchants get cautious and cancel anything that looks funny. Which is the other bad branch, the false decline. Here the order was real, the customer was real, and you just turned away their money and, often, their future business. This is not a rounding error. Across ecommerce, false declines are estimated to cost merchants on the order of $443 billion a year, versus roughly $48 billion lost to actual fraud. Retailers reject around 6% of all orders, and somewhere between 2% and 10% of those rejections are good customers who did nothing wrong.

◆ DATA The false decline is the more dangerous mistake, and it's the one with no paper trail. A chargeback shows up on a report; a cancelled good customer just quietly never comes back. Among loyal buyers, people with three or more approved orders, a single false decline is followed by a 65% drop in how often they order again, and 27% never return at all. New customers are 5-7× more likely to be wrongly declined, so the people you're paying the most to acquire are the ones the flag burns first.

Why the flag feels like a blackbox

The deepest complaint isn't that the score exists, it's that you can't see inside it. Shopify's fraud analysis weighs signals like the card's verification result, whether the billing and shipping addresses match, proxy or VPN use, the email and IP, and how closely the order rhymes with past fraud. But a good chunk of those signals stay hidden even after an order is flagged. The merchant is told the conclusion and denied the evidence, then asked to act on it with money and reputation on the line.

So they improvise. They paste the customer's email into a search bar, eyeball whether the address is real on a map, count how many times the same card has tried, message the buyer to "confirm some details," and generally rebuild, by hand, badly, one order at a time, the investigation the platform could have done for them. For a store doing a handful of orders a day, that's tolerable. For one doing hundreds, the fraud queue becomes a part-time job nobody applied for, and the temptation to just reflexively cancel every flag gets stronger by the week.

What the manual fixes miss

The usual escalation is Shopify Flow: when an order is high-risk, auto-cancel it, or tag it and hold it for review. Both are blunt. Auto-cancel hard-codes the false decline, you've now automated the act of firing good customers. Tag-and-hold just moves every flag into a pile someone still has to work through by hand, which is the problem you started with. A native rule can branch on the risk level, but it can't go and find out more. It can't check whether this email has a five-year order history with you, whether the shipping address has received twenty happy deliveries already, or whether the same device just tried four cards in ten minutes.

That's the gap. Fraud review isn't a branch on one number; it's a small investigation. And an investigation, gather the signals, weigh them, decide, and know when you're genuinely unsure, is exactly the kind of judgement you can describe in a paragraph but could never draw in a node graph.

What the automation actually has to do

The real job is to do the legwork a careful human would, on every flagged order, in seconds, and to only interrupt a person for the handful that are truly ambiguous. Written as a Dugong playbook, in plain prose, it reads like this: 

# trigger
When an order is flagged medium- or high-risk

# steps
1. Pull the signals a human would check:
   AVS/CVV result, billing vs. shipping match, email + address age,
   IP geolocation vs. shipping country, proxy/VPN, order velocity
2. Look up this customer's history with us, prior approved orders, past chargebacks, lifetime value
3. If clearly legitimate (known good customer,
   everything matches) → approve and fulfil, no human needed
4. If clearly fraudulent (mismatched everything,
   card testing, disposable email) → cancel, refund, blocklist
5. If genuinely grey → hold, send the customer a
   light verification step, and queue it for me with the full dossier
   already attached
6. Log every decision so the line between
   approve and cancel keeps getting sharper

Six lines. The compiler fills in everything underneath: the calls to each enrichment source, the lookup against your own order history, the verification email, the blocklist write, the audit log. The merchant didn't describe an algorithm, they described how they'd want a trusted operations person to handle the queue, and let the compiler decide what it had to be to deliver that.

◆ NOTE The point isn't to fulfil everything or cancel everything, it's to stop guessing on the easy ones so you can think clearly about the hard ones. Most flagged orders are obviously fine or obviously bad once you actually look. Automate those two piles and the genuinely ambiguous queue shrinks to something a human can give real attention to, with the evidence already gathered.

Why this is a compiler problem, not an app problem

There are excellent dedicated fraud apps, and for high-volume stores they're worth every cent. But many merchants don't want a second blackbox sitting on top of the first one, another score, another vendor deciding for them, another monthly fee scaled to order count. What they actually want is to encode their risk tolerance: ship to repeat customers without a second look, be strict on first orders shipping to freight forwarders, always verify when the order is over $300 and the addresses don't match. Those are sentences, not settings.

A natural-language compiler is the unlock because fraud review was never really a math problem, it was a policy problem wearing a math problem's clothes. You can write your policy down in a paragraph. You could never maintain it as a sprawling decision tree, and you definitely can't extract it from a vendor's hidden model. The flag told you something was off. The playbook is where you finally get to say what to do about it, in your own words, applied the same way every time.

The workflow worth building this week

If you've ever stared at a high-risk order and just… guessed, or worse, set Flow to cancel them all and quietly accepted the cost, this is the automation to build before any upsell flow or win-back sequence. It protects the two things the flag puts at risk at the same time: your chargeback rate, which Visa starts watching at 0.65% and which can get Shopify Payments throttled if it climbs, and your good customers, who don't get a notification when you wrongly turn them away, they just leave.

Describe it the way you'd brief a sharp operations hire on their first day: here's how we tell a real customer from a fraudster, here's when you decide on your own, and here's exactly when to stop and ask me. That's the whole brief. The compiler does the investigating; you keep the judgement for the orders that actually need it.

◆ READING If this resonates, two companion pieces: our field study on the Shopify automations no one builds, and the dispatch on the overselling problem no one automates.

If you're a Shopify merchant who's built a fraud-review playbook that finally lets you stop guessing, or you've got a horror story about a flag that cost you a great customer, the inbox is open: field-notes@dugong.live. We're collecting case studies for the next issue.