Sampling bias in entrepreneurial experiments

When a startup launches a new product, the first users to try it are rarely a random sample of its target market. On platforms like Product Hunt, where new digital products debut to an audience of early adopters, that first audience skews heavily male. Ruiqing Cao, Rem Koning, and Ramana Nanda — researchers at Harvard Business School and NBER — wanted to know whether that mismatch actually matters for a product's success.

The setup

Product Hunt is a prominent platform where makers launch new products and the community votes, comments, and tries them out. The researchers used Genderize.io to classify users by gender and found that on a typical day, roughly nine in ten active testers were men. That created a natural experiment: products aimed at women were being beta-tested by an audience that didn't represent their intended customer base.

The finding

The gap was large. Products with a female-focused target market that launched on a typical male-dominated day experienced 45% less growth in the year after launch compared to products targeting a more male audience. The mismatch between who tested the product and who was supposed to use it had a direct, measurable, and persistent effect on commercial outcomes.

The clever test

The researchers isolated the effect by exploiting natural day-to-day variation in the gender mix of Product Hunt users. On days when more women happened to be active on the platform — reducing the sampling bias for female-focused products — the gender-performance gap shrank toward zero. This wasn't just correlation: the variation in tester composition on any given day was unrelated to the characteristics of the products launched that day, giving the result a causal interpretation.

Why it matters

The study reframes diversity as an economic problem, not just a fairness one. When the people evaluating new ideas don't represent the people who will eventually use them, good products aimed at underrepresented groups get weaker signals, less traction, and less funding. The result is fewer successfully commercialized innovations for consumers who are already underserved.

For Product Hunt and similar platforms, the implication is direct: the composition of the user base is not neutral infrastructure — it is a filter that systematically advantages some products over others. For the broader startup ecosystem, the paper suggests that the underrepresentation of women among early adopters and angel investors may be quietly suppressing an entire category of viable businesses.