User Driven Design...faster

I'll be the first to admit that we have no idea what we're doing. We're learning slowly but, let's face it, there are no Cliff's Notes for building a local social network.  We had some good hunches and executed on them well enough but we'd like to be learning faster.  We are not mind readers and none of us have ran a large social web platform before.

Over the last three years we've taken on the humble task of making pothole reporting enjoyable and engaging. We have tried to structure an environment where we release every two weeks and measure accordingly.

We have played around with AB Testing before but nothing we have used has really had the power to change office culture or make us stealthier. We had been looking for a tool that would let us test our hypotheses quickly, settle tiresome inter office debates and build a more user friendly SeeClickFix.

A couple of months ago my cofounder Jeff Blasius discovered Optimizely and the team dove in quickly.  Optimizely has not only made it easy to test one interface change against another, it also makes it easy for someone on our team without web experience to modify the look, content and behavior of the site and deploy quickly and easily.
With Optimizely the new game plan is that we meet once a week to

1) Review the experiments that we were running last week.
2) Decide whether to kill, extend, expand or deploy those experiments site wide.
3) Pick the new experiments for the next week.

In the first week we are running eight experiments. A couple of the experiments are running on simple marketing pages. One experiment is on the homepage. One experiment is on an issue template page and a couple more are running on neighborhood or city homepages.
Some experiments are as simple as changing text. Some as simple as adding or removing a button to increase engagement.  In one case it includes removing a feature that countless hours have been put towards and makes us feel good when we visit our own website.

To follow is example of an Optimizely experiment and the process for running the experiment.

The Challenge: Reduce the bounce/exit rate on the SeeClickFix homepage.  By how much? By anything at all because the amount of energy that will go into the overhaul will be less time than it took to write this post.

My Hypothesis:  Simplifying the homepage will increase engagement.  Currently there is a map on our homepage that has some neat functionality that zooms around the world in near real time showing issues as they are being reported. There is also a big search bar that reads "Citizens Get Started."  Below the fold there are more links and recent blogposts and a link to our mobile app.   I want to do something that would previously have created a debilitating debate with the team. I want to drop the map that we worked so hard to make beautiful and functional. My goal is to focus eyeballs on the search box.  This is where the culture shift comes in. Optimizely is so easy and time-light that the only reasonable outcome to suggesting the dropping of the map is "lets just test it."

A little background on me before I jump into the explanation of the test: I was once a freelance web designer with strong HTML and CSS skills and limited knowledge of php and javascript.  I deployed code by ftp and have little experience with the terminal.  SeeClickFix is built in Ruby On Rails and I have never had access to a Git branch for our site.  I've been learning Javascript and the terminal in a great class at Yale aptly titled Hackyale, but I would not feel comfortable tampering with our current environment designed for the real engineers in the office.

Case Study

So here’s what happens when I open Optimizely:

           1)   I tell Optimizely what URL I want to tinker with and it pulls a view of that page into my    dashboard.

          2)   I Click on the big map that sits behind the location search bar and I select the ‘remove’ option. Goodbye Map!

          3)   I drag the search bar and its div container to a more central location on the screen and then I scale it up by 50%. Yay big search bar!

          4)   I Change the text from “Citizens Get Started” to “Report neighborhood issues and see them get fixed” I then add a little sub text that reads “Over 75K Issues fixed. Is yours next?” 

          5)   I have a little trouble picking a new background color to replace where the map was so I bounce it off a few folks in the office. We settle on a dark grey.

          6)   I name my new version and save it. I then play with a few other variations that play with keeping elements above or below the fold. For now I am not going to test those variations. Just a simple A/B test between the Current homepage and the simplified page. In Optimizely you can run a/b/c/d/e tests if you have enough traffic and time.

         7)   The next step is the scariest because it’s the most powerful and the easiest to do. It’s a big button that reads, “Start Experiment.”  Once I hit that button its off to the races. The test is live instantly and real users are proving my hypotheses right or wrong.  An even distribution of the two options is run and data is collected.  Entire time to go from idea to user testing: under 10 minutes.  If it had just been a simple text change: under one minute.

        8)   Within the hour I have enough data to tell me that my hunch might have been a good one. Optimizely is telling me that there is a 50 percent chance that my experiment will beat the baseline. It looks like engagement is increasing with the simplified page by 30-40%. Wow! All I did was kill a fancy feature and go back to basics.   Optimizely wants at least another 150 conversions before it can make a solid determination that my experiment is a winner but while I’m waiting I can create and run more experiments on other pages or line up the next experiment for the homepage if this one fails.

       9)   After a few days of running experiment “Homepage Map Drop” there is a 97.5% change that we are going to beat the baseline and we have a winner. Depending on when we look at the data the new page is increasing engagement by 25 - 30% 

Conclusion

We have only been on Optimizely for two weeks and we have run two experiments that show increased engagement over 30%. We have a few that do not have enough data yet and then we have a couple that only show a 3-5% improvement. But hey, 3-5% Improvements add up and Optimizely just bought us a whole bunch of time.   We also have an experiment that dropped a few links. It has been a point of debate for the past month. The hypothesis was wrong and engagement dropped 20% with that change, but we succeeded in answering the debate question and we can move on.

When you’re a start-up and burning cash it all comes down to how many cycles you have to experiment before you run out of the money that buys you the time. With Optimizely it feels like we have substantially slowed down time.

A couple of quirks and warnings for those looking to run experiments.

We have noticed some bizarre behavior in IE: run the cross browser tests that Optimizely provides. We also have seen some data that seems a little fishy. Without explaining too much about that make sure to check in with Optimizely support that your experiments are running properly. They have been very responsive in Twitter and email.