Chargeback Vendor Comparison Checklist: What to Verify Before You Sign

Win rates are a marketing number until you know what's in the denominator

A vendor quoting 78% wins might be counting only the disputes they chose to fight — skipping the ambiguous ones, the high-risk SKUs, the repeat-customer fraud claims that look clean on the surface. That selection bias inflates the headline and tells you nothing about how they'd perform on your actual case mix.

Before you run any vendor comparison, pull three months of your own dispute data from Shopify Admin → Payments → Disputes. Know your reason-code breakdown, your average dispute value, and your current fight-vs.-concede ratio. Every vendor claim should be evaluated against that baseline — not against their reference accounts.

Prerequisites

• Your last 90 days of dispute data exported from Shopify Payments or your processor — reason codes, outcomes, dispute values, response dates.
• A clear internal definition of which disputes you currently fight and which you concede (and why).
• At least one real disputed order you're willing to use as a test case during vendor demos.
• Your current per-dispute cost — including staff time, not just fees.

The Comparison Checklist

Case Mix and Win-Rate Audit

1. Ask the vendor to define exactly which disputes are included in their reported win rate — fought cases only, or all cases received including conceded ones.
2. Ask what percentage of incoming disputes they recommend conceding. A vendor that never concedes is burning your fees on unwinnable cases.
3. Request a win-rate breakdown by reason code — 10.4 fraud disputes and 13.1 not-received disputes have structurally different win profiles. A blended rate obscures this.
4. Ask whether their reported wins include pre-arbitration reversals, or only initial chargeback responses. These are different outcomes with different cost implications.
5. Confirm whether their case mix skews toward low-value, high-evidence disputes (easy wins) or includes high-value, mixed-signal cases like yours.

Fee Structure Verification

1. Map every fee: setup, monthly platform, per-dispute submission, success fee on won disputes, and any fees on conceded cases.
2. Calculate your total cost at three volume scenarios: your current monthly dispute count, 2x that count, and 0.5x (a slow month). Pricing models that look cheap at average volume can invert at the edges.
3. Confirm whether success fees apply to the full disputed amount or only the recovered amount — some vendors charge on the gross chargeback value regardless of what you net.
4. Ask explicitly: if a dispute is won at pre-arbitration after the vendor's initial response failed, does a second success fee apply?
5. Get the contract cancellation terms in writing. Month-to-month vs. annual lock-in changes your risk exposure significantly if the vendor underperforms.

Workflow Control and Automation Transparency

1. Ask what the vendor's automation does without human review — specifically, which dispute types trigger auto-submission vs. mandatory analyst review.
2. Verify whether you can override their fight/concede recommendation on individual cases. Some platforms lock merchants out of that decision.
3. Confirm the evidence package contents before submission — not after. You need to see exactly what gets sent to the issuer on your behalf, every time.
4. Ask whether their system reads Shopify order-level fields directly (customer IP, shipping address, fulfillment status, line items) or relies on a manual data export. Integration depth determines evidence quality on time-sensitive disputes.
5. Establish who owns the response narrative — templated boilerplate or case-specific writing. Run a real order through their demo environment and read the output rebuttal letter before you commit.
6. Confirm the escalation path: when a dispute hits an unusual pattern (reshipper address, mismatched billing country, partial delivery), who reviews it and how fast?

Auditability and Reporting Depth

1. Ask for a sample dispute-level report — not an aggregate dashboard. You need to see individual case outcomes, reason codes, submission dates, and response content per dispute.
2. Verify whether you can export raw dispute data at any time, or whether your historical records are locked inside their platform.
3. Confirm whether their reporting distinguishes between disputes won on first response vs. won after representment or pre-arbitration. Conflating these masks the real cost of their process.
4. Ask how they report on disputes they recommended conceding — are those tracked and visible, or do they disappear from the win-rate calculation?
5. Request a breakdown of their response timeline performance: what percentage of disputes are submitted with more than 48 hours remaining before the deadline? Deadline proximity is a leading indicator of operational quality.

Reference Check Protocol

1. Ask for two reference merchants in your vertical and dispute volume range — not their largest enterprise client if you're a mid-market store.
2. Ask references specifically: did the vendor's win rate on your account match what was promised during sales? If not, what was the gap?
3. Ask references whether they can see individual case evidence before submission, and whether they've ever overridden a vendor recommendation.

A real-shaped example of what vendor selection bias looks like

A mid-volume apparel merchant ran a three-vendor comparison. All three quoted win rates between 68% and 74%. After requesting reason-code breakdowns, one vendor's rate was built almost entirely on 13.1 (item not received) disputes with confirmed carrier delivery — the easiest category to win with a tracking screenshot. Their 10.4 fraud win rate was 31%. The merchant's actual dispute mix was 60% fraud claims. The headline number was accurate and completely irrelevant to the merchant's situation.

Common Mistakes

• Accepting a blended win rate without a reason-code split. Your dispute mix is not their reference account's dispute mix.
• Skipping the fee calculation at volume extremes. Success-fee models that look reasonable at 40 disputes/month can become your largest operational cost at 120.
• Not reading a real rebuttal letter before signing. Template-heavy narratives lose disputes that evidence alone would have won.
• Assuming Shopify integration means deep data access. Some vendors pull only surface-level order data. Confirm which specific Shopify fields their system reads — at minimum: Order ID, fulfillment status, shipping carrier, customer IP, billing vs. shipping address match.
• Ignoring concede-rate transparency. A vendor that never recommends conceding is optimizing for their submission volume, not your net recovery.
• Evaluating vendors during a low-dispute period. Operational gaps surface under volume pressure, not during calm months.

Revisit This Checklist When…

• Your dispute volume increases more than 30% month-over-month — your current vendor's automation thresholds may not scale cleanly.
• Your reason-code mix shifts significantly (e.g., fraud claims rise relative to fulfillment disputes) — vendor performance is reason-code-specific.
• Your vendor changes their pricing model, fee structure, or platform terms mid-contract.
• You enter a new product category or sales channel that changes your average order value or fraud profile.
• Your win rate drops two consecutive months without a clear operational explanation — that's a vendor performance signal, not just variance.
• Your contract renewal window opens. Don't auto-renew without running steps 1–5 of the fee structure section against current volume.