Showing posts with label evaluation. Show all posts
Showing posts with label evaluation. Show all posts

Friday, January 25, 2008

Listening in Multiple Directions: The Value of Tracking


Pop quiz. At any given time, do you know:
  • how many visitors are in your museum?
  • how many members are in your museum?
  • whether an individual visitor has visited before, and if so, when?
  • the purchase history of visitors buying items in your store?
Shockingly, for many museums, large and small, the answer to these questions is no. Ticket systems don't talk to membership databases. The store and the museum entrance are strangers. Each person who enters, whether they visit weekly or once in a lifetime, is treated the same.

Yes, some retail works that way. In Macy's or at the movie theater, I'm just a credit card. But not so in other facilities that encourage repeat visitation. Consider, for example, how my climbing gym uses data. When you arrive, you have to swipe your member card (comparable to museum membership) or pay for entry if you are not a member (again, comparable). The person behind the counter can instantly see how often you come, when you last visited, and can relay to you messages about your bill, classes you are taking, or warnings/compliments about behavior based on prior visits. It doesn't matter whether I've never met the person behind the desk or not--he or she can engage with me appropriately based on data. This kind of tracking enables faceless visitors to be treated as regulars, long-lost buddies, and or potential sales opportunities.

And on the web, this data use goes to the next level. Many months ago, there was a provocative post on the O’Reilly Radar called “If Google Were a Restaurant." In it, the author explored the Google's "pervasive culture of measurement," and considered how similar tracking and alterations might be made in the real world. Imagine a restaurant that tracks every single order, how much food is left on every plate, and uses that information to buy, market, and plate food differently. Imagine book stores that rearrange displays every day based on the sales of the previous day. Imagine museums that… well, we’ll get there.


The point is that Google monitors every single transaction on its site. It knows how many people in what geographic areas search for which terms when. It knows what people are most likely to click on for a given search. And it uses that data to prioritize content—to make the search functionality better.


Many interactive sites do this. Amazon tells you not only what product you might like, but when you click on a product, it tells you what people who clicked on it actually purchased. There are sites that change their inventory, their content, and their marketing to personalize your experience based on passive monitoring of your usage.


Is this Web 2.0? Absolutely. It’s not user-generated in the typical sense, but it fits the basic law of "architecture of participation with network effects": the services get better the more people use them. If Google wasn’t collecting information about what people click on for a given search term, they would have to hire thousands of people to prioritize the content on the web for meaningful results—that is, they’d have to curate the internet. Instead, Google lets users do this curating with their everyday searches for climate change and Paris Hilton. It is user-generated. It just doesn’t take any special effort on the part of the users.


So what about museums? Through the ubiquity and ease of web statistics tracking programs, museums are learning who does what where on their websites. Simply making your collection available for browsing in a database fashion—without any fancy tagging interactivity—can give you data about what parts of the collection are most interesting and or accessible to your web users. These kinds of stats have been used to help museums reorganize their sites; for example, to put PLAN A VISIT front and center when they determine that that’s the content most web visitors want.

On the web, it’s easy to track user actions, and, if you’re inclined, to act on them. But what about in the museum? As discussed in the O’Reilly post, it’s much harder to track and assimilate data in the real world than on the web. We can certainly do it at the ticket counter. Inside the museum, it becomes trickier; exhibit tracking requires human intervention or highly integrated technology. This is what evaluators do, though their monitoring requires resources and effort. Many educators hand out surveys at the end of programs. As more museums, particularly science and tech museums, move towards personalizing the museum visit using RFID and other technologies, these same technologies are being used to see how many people spend how much time interacting where. There are some privacy and data integration challenges, but real-time tracking of activity in the museum is possible.


And tracking isn’t the real challenge when it comes to “googlizing” the museum. It's not brain surgery to create a system, like that at my gym, that gives you historical information about each visitor that walks through the door.
In fact, there are many museums, including my own (The Tech), that collect data on visitor use of exhibits incidentally--but do nothing with the data. Now that RFID and barcode scanners are becoming more popular as tools to "personalize" the museum experience, some museums are generating a lot of data about who uses what exhibits for how long. And while we hand that back to visitors--in certificates congratulating them on completing several exhibits (Sony Wonderlab) or personal websites with images from their visit (The Tech)--we don't use it internally.

Why don't we get this data, and if we have it, why don't we use it? Because the challenge isn't tracking: the challenge is to listen to and act on the tracked data. That's the problem I've seen with educational program surveys--the results are tallied, the great quotes are emailed, but intelligent actions aren't taken. Let’s say you performed an evaluation in which you placed tape recorders in a museum wing for a month and recorded every single thing said. If the transcription revealed that visitors were confused, disappointed, or disaffected by exhibits, were skipping some in favor of others, how would you react? Would you repeat the experiment on other areas? Would you move the most popular exhibits to the front of the wing? Would you reconsider inclusion or implementation of the least popular ones? Or would you destroy the tape?


I think most of us would destroy the tape (or at least hide it in a drawer somewhere). On the web, we’re willing to reorganize content to improve the visitor experience. But in the museum, it’s more expensive, more painful, more personal. This is one of the secret sources of resistance to 2.0: it requires listening to people we're not used to listening to. Beth Kanter wrote about this recently, commenting:
The premise is that listening must become a priority in order to use the Web2.0 tools successfully. I think it is a pretty critical marketing practice despite what technology tools you are using.
And if we can get over ourselves and start listening, there are real financial and visitor rewards to be realized. Google and Amazon doesn’t track user actions to "appreciate" their users' interests. They do it to be more effective institutions. They do it to sell more stuff.

How can museums become more willing listeners? By designing for modularity. By acknowledging unsolved design problems (exhibits stuffed into hallways, nooks of random growth) and considering visitors' input to find solutions. By taking on more experiments so that we're generally more change-positive. By opening exhibits before they are finished so that evaluation isn't an onerous afterthought. By generating more data about visitor actions and analyzing it.

Good operations officers know how much specific exhibits cost in maintenance and disposables. Good marketing directors know that there's a direct relationship between number of visits and likeliness to buy or renew a membership. Why aren't we tracking the data that can make us more successful? Wouldn't you like to know what value different exhibits and programs have as you work out the equation of what's worthwhile and what direction to take?

Thursday, October 18, 2007

Human + coLAB Experiment Post Mortem

Thanks to all who visited the coLAB and participated in the Human + collaborative experiment over the past ten days. For those who didn’t see it, this project was an open conversation about development of a planned traveling exhibition on human enhancement technology (Human +). The exhibition is being developed by the New York Hall of Science, and I worked with Eric Siegel, the Director of NY Science, to initiate this project. The project is powered by a free software called Voicethread. To view the conversation, turn on your speakers and click the play button below. There’s about 30 minutes of content here, but you can flip through the slides and voices as you see fit.




In terms of numbers, this collaboration was a success. 206 unique people from 131 cities all over the world viewed the site 358 times. (See map on right for distribution.) There were 54 comments made by 17 people. It was blogged by three sites, including Beth Kanter of the highly regarded non-profit social media site Beth’s Blog.


Logistically, it was simple. It took Eric and I about 2 hours each to get the site up and running (content plus distribution plan). We each spent another 2 hours throughout the week checking in on the voicethread and responding to comments. There were no financial costs. There were no problems with spam or inappropriate comments. This was an unmoderated experiment, though I did add additional slides halfway through the experiment to add more venues for contribution.
But impact is what really counts.

Here are some observations from this experiment, gleaned from my impressions and yours:


A lot of you like this technology. Several people were impressed by the sound quality, the personal nature of voice, and the ease of use, and a few indicated that they would use Voicethread in their own institutions. Some of you were more fascinated by the technology’s demonstration than the specific content (which is fine!).


Participation was high. On this blog, about 0.5% of people who read a given post comment on it. On the voicethread, 8.5% who viewed it made comments, and many came back a second time to see how it had evolved. The participants were diverse, ranging from museum exhibit developers to NPR accessibility engineers to content experts to e-learning professionals. There was some emergent behavior where content experts previously unknown to Eric or me offered their support to the exhibition.


There was an inverse relationship between time of first view and participation. Participation dropped significantly after the first four days. The conversation reached a critical mass of participants quickly. After that point, many people emailed me to comment that it felt unwieldy, or that they perceived it as something already completed. It's hard to browse through lots of audio. As one person said, “it felt like watching a disjointed play.” It seems that there’s a sweet spot where just a few people have contributed to the conversation and you feel like it’s open to you. Too many and it feels overwhelming or like your contribution is not needed. It’s easier later in the process to look at the voicethread and feel like enough has already been said—thus promoting lurking over participating.


The content was interesting, but not always what was asked for. Some (including the creators of the technology) found it varied and fascinating. But there was no easy way to spin off individual “threads” of conversation on a single slide, so a divergent (interesting) point brought up by a couple people became hard to follow. The content stayed fairly surface-level, though many interesting comments, both personal and professional, were contributed.
-The purpose wasn’t totally clear. While Eric and I actively responded to other contributors, I think we could have done better to give people explicit challenges or goals so they could apply themselves concretely to solving a problem. The problem given, related to collaboration, was somewhat open-ended and proved less appealing than the Human + controversies themselves.

There was no clear way to identify the people speaking, except via their name, image, and voice. A few people commented that it would have been nice to see some basic information about speakers’ expertise and professional interest in the topic. I also would have liked an update function where people (myself included) could be notified when a new comment was added to the stream.


I left the experiment with a few core questions:

  • How can we encourage sustained participation throughout the life of a project, rather than just at its outset? How do we encourage new users to join partway through?
  • How can we guide collaboration towards a goal? What’s the balance between inviting people to talk about what they want versus what you want?
  • What platforms or technologies humanize rather than dehumanize the process?

What are your questions or comments? I look forward to doing more experiments with other technologies in the future. If you or your institution wants to get involved, let me know.