Data Driven Decisions
By now most of us have heard concepts such as the wisdom of crowds or A/B testing but still so often we make decisions without gathering data. Admittedly not every decision we make during our busy days requires data analysis but the ones that matter such as your product’s UI redesign, a price change, or advertisements often get the same treatment as your choice of lunch sandwiches. Perhaps you or someone on your team claims such connection with customers or product expertise as to not require testing. Don’t believe this!
Allow me to share with you an antecdote from my past that shows differently. Some of the facts of this story have been obfuscated to protect intellectual property but the gist of it remains true. The company that I was working with sold a product that allowed customers who purchased it to receive a return on their investment in a variable amount of time depending on how they configured the product. When getting up to speed on the product I asked everyone from the CEO to customer account managers, many who had been working with in this field for years, which was the optimal configuration. Everyone suggested a particular configuration. Being a bit of a stats geek from my days in Six Sigma I grabbed some data and started analyzing. The initial results shocked everyone because they indicated the exact opposite of the “optimal configuration”. After a complete A/B test the company ended up building a practice and product around the new ideal configuration, a big win for customers.
Ian Ayers in his book Super Crunchers offers several examples of random testing from companies as diverse as Monster.com to Capital One that have resulted in tens of millions of dollars of increased revenue. Companies such as Offermatica and Google offer A/B or even multivariate testing. Ian actually used this same techinque through online advertisements to determine the title of his book. Tim Ferriss in his book Four Hour Work Week did something very similar and recommends this approach to quick testing with advertisements for everything from business ideas to homepage redesigns.
While we caution against analysis paralysis there is a middle ground. Our mantra for processes is “Right Time, Right Process” meaning you need the process that fits best today for the your team and for the task. As we state in The Art of Scalability “Each and every process must be evaluated first for general fit within the organization in terms of its rigor or repeatability and then specifically for what steps are right for your particular team in terms of complexity.” The bottom line is, for decisions that matter get the necessary amount of data to make the best decision.