Thursday, February 04, 2010

Quick Hit: Five Evaluation Reports on Participatory Projects

What's the best way to share information about your experiments--what worked and what didn't? Publish. I want to share a few fabulous evaluation and research studies that have greatly informed my work (and specifically, the development of The Participatory Museum, which is going to the printer this weekend).

Note: these are all downloads, which means when you click on the link it will ask you if you want to save a document.
  1. The Catalyst for Change social impact study from the Glasgow Open Museum. This report, published in 2002, chronicles the history and impact of an extraordinary institution founded in 1990 to make museum collections available for use by community members for their own purposes. As founder Julian Spaulding said, "The intention was to deliver what people wanted rather than what the museum thought they wanted or what the museum thought they ought to want."
  2. The evaluation of the first Turkish Living Library, held in 2007 at the Rock for Peace festival in Istanbul. The folks who ran this event required every single participant to fill out an evaluation form, and this report demonstrates their overwhelming enthusiasm for the program as well as the tremendous social learning outcomes. The comically positive graph at the top of this post comes from this report.
  3. Dialogue in the Dark visitor and guide impact studies. This amazing "all dark" exhibition, which visitors experience in groups led by blind guides, has reached millions of people around the world. These impact studies demonstrate its incredible power both as a memorable visitor experience and as a positive impact on blind guides' social and professional lives.
  4. OK, this isn't one download, but I want to honor the Exploratorium, which publishes all of their research studies for public download here. One of my favorites that I've frequently returned to is Sue Allen and Josh Gutwill's 2004 paper, "Designing for Interactivity: Five Common Pitfalls." The content is relevant to all kinds of museums, not just science centers.
  5. A paper on "Evaluating participatory, deliberative, and co-operative ways of working," put out by the InterAct group in England. This useful and easy-to-understand paper spells out several considerations of planning evaluation for participatory projects. Most of the examples are from British civic projects, but there's lots of useful information and frameworks for projects in any field.
There are many other fabulous reports out there to learn from, but there are far more languishing in file cabinets. I was a bit surprised as I worked on the book at how often I could get access to a evaluation report with some sleuthing and asking--and how infrequently those reports were publicly available in any form. Please, if you go to the time and expense of evaluating your institution's projects, find a way to share what you've learned.

5 comments, add yours!:

Sam Ward said...

Thanks for sharing these. I've always found designing an effective evaluation plan one of the most challenging aspects of a project - but I'm always eager to learn from people who do it well.

And congrats for getting to the printer!

Ramon said...

Thank you for publishing this. It is very useful.

We invite you to have a look at the process we have started at Citilab (http://citilab.eu), which explores the relationship between curatorship, users and content around exhibitions. We have started a project exploring different relationships among these in the area of Science and Technology exhibitions. We are specially interested in the evaluation dimension of participatory processes of this kind.

Here you can have a look about how the project started with a partipatory desing of a whole exhibition about Internet.

http://expolab.net/eng

In this blog we update what users and designers are currently doing with this idea.

Peter Linett said...

Ramon, thanks for pointing us to Citilab and its cousin, Expolab: these look like wonderful initiatives.

Nina, very glad to see these reports, particularly the Interact Group paper on how to think about evaluating these experiences. But the Glasgow evaluation relies to an unfortunate extent on the appraisals of the people responsible for designing, overseeing, and implementing the program. The methodology page is vague about how many actual participants were interviewed, and we're not told whether what those users said about their experiences was similar to what the professionals said. So this is (mostly) a second-hand evaluation, and in such cases it's very hard to know what's an actual outcome and what's a projection of the wishes and values of the professionals.

Marti said...

Another source for evaluation reports of various museum, media and informal science education related projects is www.InformalScience.org where you can search and download over 200 evaluation reports (formative, front-end and summative).

Also, we are beginning to see some "participatory evaluation" approaches emerging in the design process where all stakeholders, including staff and end users/visitors become genuinely involved in establishing the criteria for success and assessing outcomes.

Anna said...

Thanks for these great links. I found the InterAct paper particularly useful and wonderfully written. The participatory project I am working on concerns workplace learning in hospitals - so not museums - but there was lots to learn from in that paper.