skills/experiment-brief/SKILL.mdView on GitHub
This applies before every bet, large or small. It produces the brief that makes the readout honest, because the success number was written down before the test ran.
Before you start
This skill works on its own. If the person has their own marketing brain (or any folder of context files about their company), read the files listed under reads in the header first and cite them for every fact. If they do not have one, do not look for one: ask for the two or three facts this play needs and go on. Never invent a fact; a blank is a question for the person.
Tools: this play speaks in verbs (check the CRM, pull the call, schedule the post). TOOLS.md at the root of this repo lists the tools recommended for each verb. The company’s own 1-context/stack.md wins when it exists. No tool is required; a spreadsheet and a mail client cover most of it.
The play
- Read the context (see Before you start). Write the hypothesis in one sentence: if we do X for audience Y, metric Z moves to threshold T within time U.
- Pick the one metric from
3-strategy/funnel.mdthe test is judged on. A test that will be judged on three metrics is three tests or none. - Set the threshold and the time box before the budget: the number that means scale, the number that means kill, the date of the readout. Default to the shortest window that can show the signal.
- Set the budget and the kill criteria: the spend or time at which the test stops early if the leading indicator has not moved.
- Name the owner, the decision this test informs (what gets more budget, what gets cut), and who makes it.
- Check
5-current/calendar.mdfor conflicts: a test that straddles a launch, an event or the dead zone will not read cleanly. - Log the brief in
decision-logwith the readout date. Schedule the readout throughexperiment-readout.
Use references/brief-template.md for the form.
What good looks like
- The best operator writes the kill number first and the budget second.
- The mediocre version launches the test and decides what success means when the numbers come in.
- Good output is a brief a stranger could grade without asking the owner anything.
Rules
- MUST state one metric, one threshold and one readout date before the test starts.
- MUST log the brief in
decision-logbefore any spend. - NEVER change the threshold after the test starts; log a new brief instead.
- NEVER run a test with no kill criteria.