Short answer: I learned to do research blending four sources of data-- existing data, qualitative, surveys & AB test results to develop consumer insight about how to delight customers.
Great story! Thanks for sharing. I once had an experience of getting data (user interview + survey) that told us not to proceed with an initiative. CEO wanted to launch it anyway. He was right. The feature was a success. People's discourse is often very different from action. We, humans, are quite irrational indeed.
Hey Gib, this was a great read. Surely learnt a lesson or two about data discovery. I have two questions, and would love for you to answer them. One, how did we establish merchandising metric as the proxy for retention? Did we have correlation data to support this? Second, what were the levers that contributed to the YoY growth of the merchandising metric till 2008, given the hypothesis Movie finding tools improve retention was proved wrong. I look forward to hearing from you :)
Hi: 1. How establish the merch metric as proxy for retention? The idea was that we should make it easy for folks to find movies they love. During DVD era, that meant making it easy to find movies members could add to their movie list. We looked at lots of potential proxy metrics (for retention) but landed on % of members who add at least 6 DVDs to their movie list in the course of a month. We made it easier to add movies to their list and, over time, retention improved. But I don't think we ever set up an AB test to prove it improved retention -- we took it on faith. 2. The things that improved the % > 6 titles in movie list were things like, "Back of Box" which was a pop-up layer that gave a summary of the title before they clicked on it. We also got better at merchandising best choice titles on the homepage. Also, we substantially improved search.
Great story! Thanks for sharing. I once had an experience of getting data (user interview + survey) that told us not to proceed with an initiative. CEO wanted to launch it anyway. He was right. The feature was a success. People's discourse is often very different from action. We, humans, are quite irrational indeed.
exactly!
Hey Gib, this was a great read. Surely learnt a lesson or two about data discovery. I have two questions, and would love for you to answer them. One, how did we establish merchandising metric as the proxy for retention? Did we have correlation data to support this? Second, what were the levers that contributed to the YoY growth of the merchandising metric till 2008, given the hypothesis Movie finding tools improve retention was proved wrong. I look forward to hearing from you :)
Hi: 1. How establish the merch metric as proxy for retention? The idea was that we should make it easy for folks to find movies they love. During DVD era, that meant making it easy to find movies members could add to their movie list. We looked at lots of potential proxy metrics (for retention) but landed on % of members who add at least 6 DVDs to their movie list in the course of a month. We made it easier to add movies to their list and, over time, retention improved. But I don't think we ever set up an AB test to prove it improved retention -- we took it on faith. 2. The things that improved the % > 6 titles in movie list were things like, "Back of Box" which was a pop-up layer that gave a summary of the title before they clicked on it. We also got better at merchandising best choice titles on the homepage. Also, we substantially improved search.