Can You Make A/B Testing Part of Your Practice?

One of the things that fascinates me about comment boards is the extent to which design impacts visitor contributions. Each of us (and every visitor) has within us the capacity to be both profound and banal, and our choice at any moment depends not just on how we feel intrinsically but also what external prompts, tools, and motivations are provided.

It's not surprising that design impacts behavior, but many people want proof that visitors are capable of more than writing "I was here" in a comment book. How do I know that design impacts visitor participation? I've seen it in project after project. I've seen how a typewriter can silently encourage people to write letters. I've seen how a "bathroom wall" can garner graffiti.

But the simplest evidence I have for the statement that design impacts visitor contributions comes from a formative evaluation performed at LACMALab for their nano exhibition in 2004. In the report, evaluator Marianna Adams described a simple experiment in visitor response. LACMALab took one question--"What connections do you see between art and science?"--and created two ways for visitors to respond. In March, visitors were offered white 4"x6" notecards and golf pencils. In April, these were replaced with blue hexagonal cards and full-size pencils.

What did they find? From the report:

The percentage of "unrelated" responses for this question decreased from 58% (with the white cards) to 40% (with the blue cards), and "specific" responses nearly doubled, increasing from 28% (with the white cards) to 50% (with the blue cards). These findings strongly support Hayes (2003) research that while the question itself has an important effect on the quality of visitor responses, the physical design of the response areas plays a prominent role in eliciting richer responses and decreasing unrelated ones.

Does this mean that visitor response stations should always use hexagonal blue cards and full-size pencils? Of course not. This finding suggests that giving people unusual or special tools can increase their dedication and focus on the task at hand. Other studies comparing regular pens and silver pens have had similar results.

This kind of experiment is called an A/B test. The museum compared visitor behavior in setup A to that in setup B.

Most museum prototyping does not follow an A/B model. We test one thing, learn from how visitors respond, and (hopefully) reiterate for the next round. This may make sense if you are trying to see how someone explores a space or approaches an activity, but it's not nearly as useful as A/B testing if you're trying to figure out how to write a great label or design a good question for visitor response.

I use A/B testing all the time to write questions for visitor comment. I've been amazed to learn that "what's the best job you've ever had?" is a lousy question but "what's the worst job you've ever had?" is a fabulous one. I'll frequently test up to ten different questions around a single exhibit. It's easy to quickly determine that some questions really are better than others in terms of prompting desired visitor response.

Here are three reasons I want to encourage you to consider A/B testing in your next experiment:

1. It forces you to set priorities for what makes a "successful" project or visitor experience. When you compare different behaviors, you will naturally express preferences for one outcome over another, and these preferences can help you understand what you value and consider to be a "good" project.
2. It helps you communicate about what you've learned with others. When you mount an exhibition and study it, the typical report is a matter of degrees--how much did people like it, how long did they stay, etc. Unless your institution has clear marks of success (i.e. more time with the object is always better), it's hard to figure out where these projects fit against benchmarks. A/B testing lets you say: "X helped us accomplish our goals more than Y." This is good internally for talking with board and staff, but it's also great externally for helping advance the field.
3. It helps you make decisions that you can apply to future projects. A/B tests reveal theories that can help you make more informed design decisions, whether in ongoing development or for your next project. Instead of saying "people learned from this exhibit," you can say, "people learned more when we did X." Websites use A/B testing all the time to see how users respond to different visual styles and prompts and introduce redesigns that will be more effective at communicating desired content or prompting desired behavior. Designers put up multiple ads on Google AdWords or show users different versions of the same site and make decisions based on what's most effective.

I know there are a few museums playing with A/B testing (most notably, the Exploratorium). But I'd love to see a whole lot more, and I'd like to see museums doing it with everything from membership drives to exhibitions. These tests don't require fancy evaluative practices or expensive equipment. To my mind, we learn best as a field from A/B tests, because they allow us to compare the incomparable and glean new insights about visitor experience.

So how about it? How can you integrate A/B testing into your work?