Run powered holdout tests for any reachable audience

Design a statistically powered test for any list you can target, run it on your channel, match outcomes back, and get incremental CAC, ROAS, lift, intervals, and saturation in the app.

Direct mail, CRM, email, SMS, or any audience you can match back to outcomes.

Power sized first

Know the audience, holdout, and minimum detectable iCAC or iROAS before launch.

Balanced assignments

Use stratified random sampling so treatment and control look comparable.

Matchback results

Upload outcomes and get lift, iCAC, iROAS, intervals, and a clear decision.

Saturation curve

See how marginal CAC changes as you reach deeper into the audience.

What you get

Everything a user-level test needs, in one tool

From sizing and assignment to results and MMM calibration. You bring the audience and the outcomes. We do the math.

Power analysis & sizing

Size the test before you spend. See the smallest iCAC or iROAS you can reliably detect, or let us recommend the holdout for the audience you already have.

Stratified random sampling

Split treatment and control with stratified random sampling so the two groups are balanced on what matters and your test is well powered.

Results in the app

Upload outcomes and get incremental CAC or ROAS, lift, confidence intervals, and a clear decision. No spreadsheets, no extra tools.

Targeting depth & saturation

Score your users and trace the diminishing-returns curve. Know your all-in campaign iCAC and the marginal cost of one more person.

Calibrate & validate your MMM

Send the measured effect and the saturation curve straight into your MMM to calibrate the channel and validate the model.

One central test library

Every test, assignment, and result is stored by brand. Search it, reference it, and reuse it as evidence later.

How it works

From a list to an incremental CAC, all in the UI

Four steps, no spreadsheets, nothing extra to wire up.

1

Upload your audience

Bring the list of users you can reach. Add an optional 0 to 100 score for each person to unlock depth analysis.

2

Size and split

Run the power analysis, then split treatment and holdout with stratified random sampling. Download the assignment to launch.

3

Run and match back

Send to the treatment group on your channel. Upload outcomes and we match exposed users back to what happened.

4

Get results and calibrate

See incremental CAC or ROAS, lift, and the saturation curve in the app, then feed them into your MMM in a click.

Test design

Balanced groups, by design

A coin flip can hand you a lopsided test. We split your audience with stratified random sampling, so treatment and holdout look alike on the things that drive your outcome. The result is a more powered test and a cleaner read.

Balance on value and score

Stratify on revenue, propensity, or any pre-test score so high and low value users are evenly represented in both groups.

Balance on segments

Stratify on region, customer type, lifecycle stage, or channel so no segment skews one arm of the test.

Reproducible and auditable

Every assignment is reproducible and logged, so you can stand behind the split.

Balanced within every segment

High value
Mid value
New / low value
Reactivation
Treatment Holdout

Diminishing returns

Users targeted, best firstIncremental

Campaign iCAC

all-in, across everyone you reach

Marginal CAC

the cost of one more person

Pre-test scoring

Score your users. Find your depth.

Give each user a 0 to 100 score for propensity, value, or whatever you model, and targeting depth becomes a dial. Target your best users first. Go deeper and each additional person costs more. We trace the whole diminishing-returns curve from a single test.

Find the right depth

See your all-in campaign iCAC and the marginal CAC of reaching one more person, so you know exactly how far down the list to go.

Calibrate your MMM

That same curve calibrates your MMM's saturation, and the headline iCAC or ROAS calibrates the channel. One experiment makes the model smarter.

Channels

If you can match it back, you can measure it

User-level testing works on any channel where you control the audience and can match exposed users back to outcomes. Person, household, or account level.

Direct mail

Prospecting, win-back, and reactivation. Use clean treatment and holdout files to measure real incremental response and profit, not attributed conversions alone.

CRM, email, SMS, push

Holdouts on promotions, lifecycle journeys, and offers. Measure revenue and conversion by assignment, with guardrails on fatigue and unsubscribes.

Any audience-targetable platform

Build the segment, target the treatment group, and match conversions back by id. If you can upload an audience, you can test it.

Direct mail, CRM, or paid social. Person, household, or account. Any reach with a matchback works.

Measurement Roadmap

Calibrate and validate your MMM

Every user-level result becomes evidence for the MMM we are launching next, calibrated at the exact point in time you ran the test.

Calibrate the channel

Lock in the measured incremental CAC or ROAS as the channel's effect, straight from a real holdout.

Calibrate the saturation

Feed the diminishing-returns curve in so the model learns where the channel starts to saturate.

Validate the model

Check that your top-down MMM agrees with what the holdout actually measured.

One experiment, a smarter model.

Your test database

Every test, in one place, by brand

Designs, assignments, and results are stored per brand. The app runs every calculation for you, keeps a full history you can search and reference, and turns published results into reusable MMM evidence. No spreadsheets to maintain, nothing to wire up.

One source of truth

All your user-level tests and results live together, scoped to your brand and ready to reference.

Calculated for you

iCAC, iROAS, lift, confidence, and the saturation curve are computed in the app on every run.

Reuse the evidence

Publish a result and it is ready to calibrate and validate your MMM, now or later.

Human in the loop

Built for everyone on the team

You don't need to be a statistician. The hard math runs under the hood; you point, click, and make the call with the full picture in front of you.

MarketerAnalystCMOData ScientistVP of MarketingDirectorGrowth ManagerBrand ManagerAgencyFounder
The measurement triangle

One framework. Every method talks to the others.

Geo, user-level, and platform tests all feed one central database. That shared truth calibrates and validates your MMM, while attribution keeps everything pointed in the right direction. Run any piece on its own — or run them together and let each one make the others stronger.

Inform. Attribution gives the MMM a fast, directional read on what's working between tests.

Calibrate. Incrementality tests anchor the MMM to causal ground truth, not just correlation.

Validate. Holdout tests check that attribution and the model agree with what really happened.

See the full framework
FAQ

User-level incrementality FAQ

New: AI Marketing Scientist

Or just ask your AI.

Connect any AI agent through MCP/API to read-only user-level test context: audience assignments, stratification choices, matchback results, lift, iCAC/iROAS, intervals, and saturation curves. Shako Stats computes the deterministic readout; the AI helps your team understand what it means.

Meet your AI Marketing Scientist

One-click OAuth for Claude & ChatGPT, read-only, anyone on the team can ask

Run your first user-level test.

Size it, split it, measure it, and calibrate your MMM in one place.

Paid B2B plans built for brands & agencies.