We Shipped an AI Feature Users Loved. The A/B Test Told Us to Delete It.

You built an AI feature. Everyone loved it in the demo. Should you ship it?

At GetYourGuide we added an AI-generated review summary to our Android app's activity page, then A/B tested it in Statsig before rollout. Seven million users later, it had reduced conversion by ~0.8% a statistically significant loss. The reason hid in a secondary metric: people read the reviews ~10% less.

This lightning talk walks one real experiment end-to-end from an Android engineer's seat. You'll learn:
- Why experiments are your safety net on Android, where there's no instant rollback
- How to wire an experiment into a Compose + MVI screen: variation in a reducer, cold-start defaults, exposure on view
- How to read results honestly and why the "mechanism metric" told the real story
- The peeking trap that nearly made us ship the wrong thing
- Why a losing test is your next hypothesis, not a dead end

You'll leave able to run an experiment you trust and to tell a feature that works from one that wins.