In the book, on page 255 he relates a story about testing they did for a Flash-based shopping cart. It’s in the chapter about testing (A/B, multivariate, etc.). He says, “…Yet when we tested it live on our website by using an A/B test, the Flash cart initially performed terribly. This was quite a shock. It turns out customers were too used to the more-complicated and ‘cumbersome’ HTML experience and not ready for an optimized cool experience.”
As I read it, this story worked its way into my brain and rattled around a bit. It made me think, “But was it still the wrong idea?”
What I’m getting at is that with anything there is a learning curve. If we see the results of a test like this and say, “Well, they are happier with the more “cumbersome” way of doing things…we’ll just keep it that way,” will we miss an opportunity? I mean, if the Flash-based is TRULY the better way to do it, don’t we do a disservice to the customer by not making the change and providing training? Sure, it may be a difficult transition, but ultimately it will be for good.
I don’t necessarily mean this is the solution for his story because a) Avinash is FAR smarter than I am and b) I’m sure he made the right choice (whatever it was). The story he related was more the catalyst for my thoughts.
My question is, how do we manage the tension between doing what we KNOW will ultimately benefit the customer and what they are comfortable with? If we simply stuck with that which made the customer comfortable, nothing new would ever happen. At what point do we look at the data and say, “Yes, I see what the customer is happier with, but I KNOW that when they learn this new way it will be better for them in the long run.”
I don’t know the answer, but it’s certainly worth pondering.