How to run a warmup pilot that proves something

What a pilot can settle, and what it cannot

A pilot is not a trial. A trial is a billing arrangement. A pilot is a written plan that fixes, in advance, what you will change, what you will leave alone, what you will measure and what result would make you stop. The plan is what turns a number into an argument.

Keep the four jobs separate while you design the pilot. Warmup sends traffic. A placement test measures one send. Authentication is a configuration. List verification cleans addresses. A pilot that changes two of them at once can conclude nothing about either.

The protocol, written before anything is connected

Eight lines. Write them in a document, date it internally, and do not edit them once the pilot starts. If a line has to change, the pilot restarts.

1. The question. One sentence, answerable by yes or no. Not "does warmup work", but "do the six warmed mailboxes reach a different folder split than the six untouched ones, on the same message".
2. The unit. Mailboxes, not domains and not campaigns. Warmup services are sold per mailbox, so the pilot counts in mailboxes too.
3. The split. Half the mailboxes warmed, half untouched, drawn at random and on the same sending domain and provider. The untouched half is the hold-out, and it is the only part of the design that carries the argument.
4. The frozen variables. Same subject line, same body, same list, same cadence, same sending platform, in both groups, for the whole period. Anything you cannot freeze goes in the limits section of the report.
5. The measurement. One placement test per mailbox at the start, one at the end, on the same seed list, plus the bounce log and the DMARC aggregate reports, which cover real recipients and cost nothing.
6. The duration and the stop date. Two to four weeks is the usual vendor recommendation for a warmup ramp. Fix the date before you start, so a disappointing week cannot quietly extend it.
7. The stop rule. The condition that ends the pilot early: a bounce rate above your threshold, a complaint spike, an authentication failure, or a blocklist entry. Write the number.
8. The access plan. What each account gets, who grants it, and the exact revocation step, written before anyone clicks Authorize.

Six mailboxes per arm is a practical floor for an agency pilot, not a statistical one. With small groups, only a large gap means anything, and you should say so in the report rather than quote a percentage point difference.

Why the hold-out group is the whole argument

The failure mode is always the same. A team measures placement, switches on a warmup service, measures again three weeks later, sees a better split and writes that warmup improved deliverability. That sentence is not supported by the measurement. Between the two runs, at least five other things moved.

The hold-out does not remove any of these. It exposes them. Both groups live through the same ageing, the same filter updates and the same seed list churn, so whatever moved in both is not your tool. Only the gap that opens between the groups belongs to the variable you changed, and even that gap is an association, not a mechanism.

If a hold-out is impossible, because you only have two mailboxes and both need warming, say so and stop claiming a result. Run the pilot as an operational test instead: does it connect, what does it ask for, what does it export, what does it cost across every mailbox. Those questions have honest answers without a control group.

A fictional pilot plan you can copy

The plan below is fictional and written for this guide. The figures are invented to show the shape of a protocol. They describe no product, no customer and no pilot we ran.

The instrument, chosen before the pilot and not during

Pick the measuring tool first and keep it for the whole period. Changing instruments halfway makes the two readings incomparable, in the same way that changing the seed list does.

GlockApps sells inbox placement and spam testing as its own product, with plans counted in test credits and a free tier of two tests, which is enough to rehearse the protocol on a single message. MailReach bundles a lighter spam tester next to its warmup service, which is convenient and also means the same vendor supplies both the treatment and the measurement. That is a reason to prefer an independent instrument when the pilot has to convince someone else.

Whatever you choose, run the repeat control first: the same seed list, the same message, twice, 48 hours apart, nothing changed in between. If the two runs disagree widely, your instrument cannot see a difference smaller than that noise, and the pilot has to be sized accordingly or abandoned.

Access, permissions and the revocation you plan on day one

A warmup service needs to send and read mail as you. That is a real grant of access to a mailbox, and it outlives the pilot unless you remove it. Decide the shape of the grant before the trial, not after.

Two connection shapes exist in practice. The first is an authorisation flow, where you sign in with your provider and approve a set of permissions for the vendor. The second is a set of sending and reading credentials, typically an application specific password with a server address and a port. Which one a given vendor offers, and which permissions it requests, changes over time. Read the vendor's own connection documentation on the day you connect, and record the answer, because we do not restate vendor permission scopes here from memory.

• Google, application specific password. Google requires 2-Step Verification on the account to create one, and states that it cannot be created for accounts using only security keys, for accounts under Advanced Protection, or while signed in to a work or school account. Revocation is a Remove action in the App Passwords list, after which, in Google's words, "the app can't access your Google Account again". Source: Google Account Help, app passwords.
• Google, authorised application. Linked third party apps are listed in the Google Account and can be reviewed or removed at any time. Google is explicit about the limit of that action: removing access stops future access, and "you may need to contact the developer of the app to request that they delete the data they already have". Source: Google Account Help, third party access.
• Microsoft and other providers. The equivalent screens exist but the paths differ by tenant and by admin policy. Find and test the revocation path in your own tenant before the pilot, and record it in the plan. We have not verified those paths for every tenant configuration, and we do not publish a path we have not opened ourselves.

Revocation is not deletion. Warmup traffic, message samples and metadata already collected stay with the vendor until it deletes them. If that matters for a client mailbox, ask for the deletion procedure in writing before the pilot, and keep the answer with the plan.

One more rule for agencies: never pilot on a client mailbox without written approval that names the tool, the access granted and the removal date. A pilot is a change to the client's mail system, and it belongs in the change log you hand back.

Exports, taken out before the trial ends

The data you cannot export does not exist once you stop paying. Before the last day of the pilot, take out the raw material of your own report.

1. The placement runs. Per provider group, per mailbox, with the seed counts, not only the summary percentage. A percentage without its denominator cannot be recomputed later.
2. The authentication results recorded at each run, so a later disagreement about what was configured has an answer.
3. The warmup activity as the vendor reports it: volumes sent per day and per mailbox. It is the only trace of what the treatment actually was.
4. The bounce and complaint logs from your sending platform, which belong to you and cover real recipients rather than seeds.
5. The invoices, per mailbox and per credit pack, so the cost line of the report matches what was actually billed.

Check the export format during the first week, not the last. A dashboard that only draws charts, with no download of the underlying rows, is a finding about the tool and belongs in your report. Which formats each of the five products exports, and whether an API covers it, is not something we have tested here, so treat it as an open question to answer during your own pilot.

Write the conclusion in sentences you can defend

The last step of the protocol is the vocabulary. Three sentence shapes survive scrutiny, and three do not.

If the honest conclusion is that nothing measurable happened, that is a result, and it is worth exactly what it cost you. The alternative, an unfrozen before and after with a confident percentage, costs more later, when someone asks how the number was obtained.

Price the outcome across your whole fleet before you sign anything. Per mailbox pricing starts small, at $15 per month for the entry plan at Warmup Inbox, and is then multiplied by every mailbox and every month you keep it running.