How to Run Growth Experiments as a PM

TL;DR

• Every growth experiment needs a specific, falsifiable hypothesis, not just a general idea to try.
• Decide the success metric and sample size needed before launching, not after seeing early results.
• Most experiments should fail — that's expected, and the process is designed to learn cheaply from those failures.

Running growth experiments is how a product manager tests specific ideas for improving acquisition, activation, retention, or revenue, without committing to a full build before knowing if the idea actually works. The value of the process comes from disciplined hypothesis-testing, not from any individual experiment succeeding.

Quick facts

• A good growth experiment hypothesis follows the structure: "If we do X, we expect Y metric to change by Z, because [reasoning]."
• Most growth experiments don't produce a positive result — treating this as normal, not a failure, is essential to running the process well long-term.
• This builds on How to Reduce Time to First Value in Onboarding, a common target for growth experiments.

How to run growth experiments as a PM, step by step

1. Start from a real problem or opportunity, not a random idea. Use existing data — a funnel drop-off point, a customer complaint pattern, a competitor's approach — to identify where an experiment could plausibly move a meaningful metric.
2. Write a specific, falsifiable hypothesis. "If we simplify the signup form, activation will improve" is too vague — "If we reduce the signup form from 6 fields to 2, activation rate will increase by at least 5 percentage points, because form length is currently a top drop-off point" is testable.
3. Choose one primary success metric before launching. Deciding this upfront prevents the common trap of retroactively picking whichever metric happened to move after the fact.
4. Determine the sample size and run time needed for a reliable result, based on current traffic and typical conversion rates — running a test too briefly risks acting on noise rather than a real signal.
5. Design the smallest version of the experiment that can validate or invalidate the hypothesis. A fully built feature isn't usually necessary — a simplified prototype, a fake-door test, or a manual process behind the scenes is often enough to learn from cheaply.
6. Run the experiment for its full planned duration, resisting the urge to call it early based on encouraging or discouraging early results that haven't reached statistical reliability yet.
7. Analyze the result honestly against the original hypothesis, not against whatever result would be most convenient to report.
8. Decide the next action based on the result: fully build and ship a validated idea, iterate on a partially promising one, or discard a clearly invalidated one — and document the learning either way so it isn't relearned later by someone else.

Why most growth experiments should fail

A growth experimentation process where most tests succeed usually means the ideas being tested aren't ambitious or uncertain enough to be worth testing — genuinely uncertain, high-potential ideas fail more often than they succeed, precisely because the outcome wasn't already obvious. Treating a failed experiment as a normal, useful outcome (a cheap way to avoid a costly full build of something that wouldn't have worked) rather than a personal or team failure is what makes a sustainable experimentation culture possible.

A worked example

A PM notices from funnel data that many users abandon during a required "connect your calendar" step early in onboarding. Hypothesis: "If we make the calendar connection optional and move it to after the user's first successful action, activation rate will increase by at least 8%, because this step currently forces a decision before users have experienced any value." The team builds a lightweight version — the calendar step is skipped and moved to a later prompt for half of new signups — and runs it for two weeks to reach a reliable sample size. Activation rate for the test group rises by 11%, validating the hypothesis; the team then permanently ships the change and documents the reasoning for future onboarding decisions.

Common mistakes when running growth experiments

• Testing vague ideas without a specific, falsifiable hypothesis, making it impossible to draw a clear conclusion from the result.
• Calling an experiment early based on encouraging early data, before reaching a reliable sample size — early results are frequently misleading.
• Picking the success metric after seeing the results, rather than committing to one metric upfront, which invites cherry-picking a flattering interpretation.
• Treating a failed experiment as wasted effort rather than a valuable, cheap way to avoid building something that wouldn't have worked.

FAQ

How long should a growth experiment run? Long enough to reach a reliable sample size given current traffic — this varies significantly by product, but committing to a minimum duration or sample size before launch prevents ending a test too early based on unreliable early data.

What if an experiment shows mixed results? Mixed results usually mean the sample size wasn't sufficient, the hypothesis was too broad, or a secondary factor is influencing the outcome — worth investigating further rather than declaring a clean win or loss prematurely.

Do growth experiments always require A/B testing infrastructure? Not always — smaller teams can sometimes run simpler before/after comparisons or fake-door tests, though true A/B testing with a control group gives more reliable, defensible results when available.

How many growth experiments should a PM run at once? This depends on team capacity and traffic volume, but running too many simultaneous experiments on overlapping metrics can make it hard to attribute results to the right cause — most teams limit concurrent experiments touching the same funnel stage.