Experiments

Experiments lets you A/B test a change before you commit to it. Run a challenger — a different evidence configuration or rebuttal letter — against your current champion on a share of live disputes, measure the real win rate and dollar impact with a confidence read, and promote the winner when you’re convinced. Nothing auto-promotes — you stay in control.

How A/B experimentation works

When an experiment is running, every new dispute that matches its scope is randomly assigned to one of two arms:

Champion — your current approach (the baseline).
Challenger — the change you’re testing.

The split is set by the challenger share (e.g. 20% challenger / 80% champion). Assignment is sticky: once a dispute is placed in an arm it stays there, so a case is never fought two different ways. Disputes that don’t match the experiment’s scope are untouched and keep using your champion — so you’re always comparing like for like, on live traffic, at the same time.

DisputeDash then tracks each arm’s win rate and dollar recovery and computes how likely the challenger is to genuinely beat the champion (not just win by luck).

What “champion” means

The champion is whatever would apply if you hadn’t set up the challenger — your current default evidence config or letter for that segment. It’s snapshotted when you start the experiment, so if you edit your defaults mid-run, the test won’t drift: both arms stay fixed to what they were at the start. For letter tests this matters especially — the champion is resolved as the letter your challenger is replacing, so the two arms are always two different letters, never the same one compared against itself.

What you can test

An experiment tests one or both of:

Evidence config — a different set of evidence types for a reason category (which documents get gathered and submitted).
Rebuttal letter — a different letter template.

Letter testing is finer-grained than evidence testing. A letter is tied to a specific reason code and to physical-vs-digital, so a letter experiment only involves disputes that would actually use that letter — the ones matching the challenger letter’s exact (reason code, physical/digital) pair.

Setting up an experiment

Admins create an experiment by choosing:

the scope — a brand (or all brands), a reason category, and a processor (or all processors),
the challenger share — what fraction of matching live disputes get the challenger,
and the challenger itself — an evidence-type selection and/or a rebuttal letter.

An experiment is created as a draft. It doesn’t touch any live disputes until you start it.

Testing a rebuttal letter

To test a letter, tick Test a rebuttal letter and pick the challenger letter from the dropdown. You don’t have to leave the page to make a variant:

+ New letter — write a fresh challenger letter inline.
⧉ Duplicate selected — copy the letter you’ve selected into an editable draft, tweak the wording, and it’s selected as the challenger automatically.

Both open the same editor you’d use in Settings → Rebuttal Letters, so you can also just ⧉ Duplicate any letter there and edit the copy. A duplicated letter starts with the original’s body and snippets and empty reason codes for you to assign.

The challenger is a real letter saved to your workspace. Create it brand-specific when you want to test a variant for one brand — that’s the default from the experiment form.

Reading the results

Each experiment card shows, for champion vs challenger:

Win rate and dollars recovered per arm,
the probability the challenger beats the champion — a Bayesian read that accounts for how much data you have, so a 3-win lead on 5 disputes isn’t treated like a 30-win lead on 500,
the expected win-rate lift if you promote,
the dollars you’re giving up while testing (the cost of running the losing arm), and
the estimated days to confidence at the current volume.

Let it run until the probability-to-beat is convincing and the estimate has settled — small samples swing wildly early on.

Deciding

When you’re ready, admins can:

Promote challenger — make it your new champion. For an evidence config, the challenger’s evidence types become your default for that reason category. For a letter, the challenger’s wording is written into that brand’s letter.
Keep champion — end the test with no change to your defaults.
Stop (worse / inconclusive) — end an experiment that’s clearly losing or hasn’t separated.
Delete — remove a draft you never started (running and concluded experiments are kept for history).