I'll be the first to admit that we have no idea what we're doing. We're learning slowly but, let's face it, there are no Cliff's Notes for building a local social network. We had some good hunches and executed on them well enough but we'd like to be learning faster. We are not mind readers and none of us have ran a large social web platform before.
Over the last three years we've taken on the humble task of making pothole reporting enjoyable and engaging. We have tried to structure an environment where we release every two weeks and measure accordingly.
We have played around with AB Testing before but nothing we have used has really had the power to change office culture or make us stealthier. We had been looking for a tool that would let us test our hypotheses quickly, settle tiresome inter office debates and build a more user friendly SeeClickFix.
A couple of months ago my cofounder
Jeff Blasius discovered
Optimizely and the team dove in quickly. Optimizely has not only made it easy to test one interface change against another, it also makes it easy for someone on our team without web experience to modify the look, content and behavior of the site and deploy quickly and easily.
With Optimizely the new game plan is that we meet once a week to
1)
Review the experiments that we were running last week.
2)
Decide whether to kill, extend, expand or deploy those experiments site wide.
3)
Pick the new experiments for the next week.
In the first week we are running eight experiments. A couple of the experiments are running on simple marketing pages. One experiment is on the homepage. One experiment is on an issue template page and a couple more are running on neighborhood or city homepages.
Some experiments are as simple as changing text. Some as simple as adding or removing a button to increase engagement. In one case it includes removing a feature that countless hours have been put towards and makes us feel good when we visit our own website.
To follow is example of an Optimizely experiment and the process for running the experiment.
The Challenge: Reduce the bounce/exit rate on the SeeClickFix homepage. By how much? By anything at all because the amount of energy that will go into the overhaul will be less time than it took to write this post.
My Hypothesis: Simplifying the homepage will increase engagement. Currently there is a map on our homepage that has some neat functionality that zooms around the world in near real time showing issues as they are being reported. There is also a big search bar that reads "Citizens Get Started." Below the fold there are more links and recent blogposts and a link to our mobile app. I want to do something that would previously have created a debilitating debate with the team. I want to drop the map that we worked so hard to make beautiful and functional. My goal is to focus eyeballs on the search box. This is where the culture shift comes in. Optimizely is so easy and time-light that the only reasonable outcome to suggesting the dropping of the map is "lets just test it."
A little background on me before I jump into the explanation of the test: I was once a freelance web designer with strong HTML and CSS skills and limited knowledge of php and javascript. I deployed code by ftp and have little experience with the terminal. SeeClickFix is built in Ruby On Rails and I have never had access to a Git branch for our site. I've been learning Javascript and the terminal in a great class at Yale aptly titled Hackyale, but I would not feel comfortable tampering with our current environment designed for the real engineers in the office.
Case Study
So here’s what happens when I open Optimizely:
1) I
tell Optimizely what URL I want to tinker with and it pulls a view of that page
into my dashboard.
2) I
Click on the big map that sits behind the location search bar and I select the
‘remove’ option. Goodbye Map!
3) I
drag the search bar and its div container to a more central location on the
screen and then I scale it up by 50%. Yay big search bar!
4) I
Change the text from “Citizens Get Started” to “Report neighborhood issues and
see them get fixed” I then add a little sub text that reads “Over 75K Issues
fixed. Is yours next?”
5) I
have a little trouble picking a new background color to replace where the map
was so I bounce it off a few folks in the office. We settle on a dark grey.
6) I
name my new version and save it. I then play with a few other variations that
play with keeping elements above or below the fold. For now I am not going to
test those variations. Just a simple A/B test between the Current homepage and
the simplified page. In Optimizely you can run a/b/c/d/e tests if you have
enough traffic and time.
7) The
next step is the scariest because it’s the most powerful and the easiest to do.
It’s a big button that reads, “Start Experiment.” Once I hit that button its off to the races. The test is
live instantly and real users are proving my hypotheses right or wrong. An even distribution of the two options
is run and data is collected.
Entire time to go from idea to user testing: under 10 minutes. If it had just been a simple text
change: under one minute.
8) Within
the hour I have enough data to tell me that my hunch might have been a good
one. Optimizely is telling me that there is a 50 percent chance that my
experiment will beat the baseline. It looks like engagement is increasing with
the simplified page by 30-40%. Wow! All I did was kill a fancy feature and go
back to basics. Optimizely
wants at least another 150 conversions before it can make a solid determination
that my experiment is a winner but while I’m waiting I can create and run more
experiments on other pages or line up the next experiment for the homepage if
this one fails.
9)
After
a few days of running experiment “Homepage Map Drop” there is a 97.5% change
that we are going to beat the baseline and we have a winner. Depending on when
we look at the data the new page is increasing engagement by 25 - 30%
Conclusion
We have only been on Optimizely for two weeks and we have
run two experiments that show increased engagement over 30%. We have a few that
do not have enough data yet and then we have a couple that only show a 3-5%
improvement. But hey, 3-5% Improvements add up and Optimizely just bought us a
whole bunch of time. We also
have an experiment that dropped a few links. It has been a point of debate for
the past month. The hypothesis was wrong and engagement dropped 20% with that
change, but we succeeded in answering the debate question and we can move on.
When you’re a start-up and burning cash it all comes down to
how many cycles you have to experiment before you run out of the money that
buys you the time. With Optimizely it feels like we have substantially slowed
down time.
A couple of quirks
and warnings for those looking to run experiments.
We have noticed some bizarre behavior in IE: run the cross
browser tests that Optimizely provides. We also have seen some data that seems
a little fishy. Without explaining too much about that make sure to check in
with Optimizely support that your experiments are running properly. They have
been very responsive in Twitter and email.