Data-Driven Product Decisions: Beyond Vanity Metrics

Every startup claims to be data-driven. Most aren’t. Myles Ndlovu has built products where the right metric changed the entire strategy, and products where the wrong metric led the team in circles for months.

Vanity Metrics vs. Actionable Metrics

Vanity metrics make you feel good but don’t inform decisions:

Total registered users (includes inactive accounts)
Page views (doesn’t tell you if anyone did anything useful)
App downloads (most people never open the app again)
Total transaction volume (are those profitable transactions?)

Actionable metrics tell you something you can act on:

Monthly active users who transact at least once
Conversion rate from sign-up to first transaction
Revenue per active user
Customer acquisition cost vs. lifetime value
Churn rate by cohort

The test: if the metric goes up, do you know what to do differently? If not, it’s a vanity metric.

The Metrics That Matter for Fintech

Activation Rate

What percentage of sign-ups complete their first meaningful action? In a payment app, that’s the first payment. In a lending app, that’s the first loan application.

If activation is low, your onboarding is broken. Focus everything on getting users to their first “aha” moment.

Transaction Frequency

How often do active users transact? Weekly? Monthly? This tells you whether your product is a habit or an occasional tool. Habits build defensible businesses.

Net Revenue Retention

Are existing customers spending more over time? In B2B fintech, net revenue retention above 100% means you’re growing even without acquiring new customers.

Cohort Retention

Don’t look at overall retention — look at it by cohort. Is January’s cohort retaining better than December’s? If each cohort retains better than the last, your product is improving. If not, you’re acquiring users into a leaky bucket.

Unit Economics

For every transaction or customer:

What does it cost to acquire them?
What does it cost to serve them?
How much revenue do they generate?
Over what time period?

If serving a customer costs more than they generate, volume makes things worse, not better.

Building a Data Culture

Instrument Everything

Track user actions from the start. It’s much harder to add instrumentation later than to include it from the beginning.

At minimum, track:

Every screen/page view
Every button click and form submission
Every API call and its result
Every transaction with its outcome
Every error and its context

Make Data Accessible

If only one person can query your data, you don’t have a data culture. Give product managers, designers, and engineers access to dashboards and the ability to run their own queries.

Review Weekly

Set a weekly cadence to review key metrics as a team. Not a presentation — a discussion. What moved? Why? What should we try next?

Run Experiments

A/B testing isn’t just for big companies. Even with modest traffic, you can run experiments:

Test two onboarding flows
Test different pricing
Test different feature placements

The key is to define your success metric before the experiment, not after.

Common Data Mistakes

Averaging everything: Averages hide distribution. The average transaction size might be $50, but if half your users transact $5 and half transact $95, you’re serving two very different segments.

Ignoring qualitative data: Numbers tell you what’s happening. Talking to users tells you why. Both are essential.

Optimising too early: Don’t A/B test button colours when your product-market fit is uncertain. Optimisation matters when the fundamentals are right.

Survivorship bias: You’re analysing users who stuck around. What about the ones who left? Exit surveys, session recordings of churned users, and failed onboarding analysis often reveal more than studying happy customers.

The Decision Framework

When making a product decision:

State the hypothesis: “We believe that X will improve Y by Z%”
Define the metric: How will you measure success?
Set the threshold: What result would make you proceed, iterate, or abandon?
Run the experiment: Ship the smallest version that tests the hypothesis
Analyse honestly: Did it work? If not, what did you learn?

Data doesn’t make decisions for you. It reduces the uncertainty in human decisions. The best product teams use data to challenge assumptions, not to confirm them.