Short answer: Identify your high-level engagement metric, form hypotheses about how you will improve it, then identify a proxy metric for each of these hypotheses (e.g. your product strategies).
Love the idea of revisiting proxy metrics and not looking at them as something set in stone. :)
You mentioned evaluating performance of a product manager via proxy metric he/she is driving, wondering whether it could lead to a case where PM becomes over-investment in the metric and look for confirmation in data? If so, how to prevent it?
1. You say that eventually, we establish a causal relationship between proxy metric and high-level engagement metric. Shouldn’t we start there? Proxy metric should already be available like you mentioned. So, we should be able to establish correlation and causality already even before deciding on the proxy metric. If a team chooses a metric after debating for a few months and then few more months down the line, we discover that the proxy metric isn’t causally related to engagement metric, isn’t that a lot of wasted effort, time and money?
2. In absence of causal relationship between proxy metric and high-level engagement metric, how do I diagnose if my experiment didn’t move the proxy metric? Was the chosen metric wrong? Was my hypothesis incorrect? Any pointers?
Love the idea of revisiting proxy metrics and not looking at them as something set in stone. :)
You mentioned evaluating performance of a product manager via proxy metric he/she is driving, wondering whether it could lead to a case where PM becomes over-investment in the metric and look for confirmation in data? If so, how to prevent it?
Excellent work Gib!
Thx!
Great essay. I found it while I am thinking about the product metrics for personalizing the experience for our online grocery marketplace.
Do you think % of existing customers who add more than x products to the cart before chechout is a proxy metric for retintion ?
Great essay! Loved it. Few questions:
1. You say that eventually, we establish a causal relationship between proxy metric and high-level engagement metric. Shouldn’t we start there? Proxy metric should already be available like you mentioned. So, we should be able to establish correlation and causality already even before deciding on the proxy metric. If a team chooses a metric after debating for a few months and then few more months down the line, we discover that the proxy metric isn’t causally related to engagement metric, isn’t that a lot of wasted effort, time and money?
2. In absence of causal relationship between proxy metric and high-level engagement metric, how do I diagnose if my experiment didn’t move the proxy metric? Was the chosen metric wrong? Was my hypothesis incorrect? Any pointers?