Adventures in Evaluating Participatory Exhibits: An In-Depth Look at the Memory Jar Project

A man walks into a museum. He shares a story. He creates a visual representation of his story. He puts it on the wall.

How do you measure the value of that experience?

Two years ago, we mounted one of our most successful participatory exhibits ever at the Santa Cruz Museum of Art & History: Memory Jars. Over three months, about 600 people filled mason jars with personal memories and put them on display. Better yet, the graduate student who led this project, Anna Greco, documented the whole project and did in-depth analysis of the visitor contributions. This post shares some of the highlights from the project and from Anna's research. I strongly recommend checking out her entire thesis [pdf] if you want to know more.

THE BASICS

The Memory Jar project was simple. We filled a small gallery from floor to ceiling with shelves of Mason jars. We invited visitors to "bottle up" a memory in a jar, using craft materials to fill the jar with evocative objects and a hand-written label to tell their story. There were no written instructions, just a mural that suggested what to do and labels that prompted people for their name and memory. The project was linked to a larger exhibition, Santa Cruz Collects, about why local folks collect things. We realized that the most valuable things many of us collect are intangible--our memories--and the Memory Jar project was born.

From the beginning, we observed pretty amazing experiences happening with the Memory Jars. People were spending a long time working on them. Some of the stories were quickies, but others were powerful and personal. We started with 400 jars and assumed we wouldn't fill them all. Instead, we had to do a rush order on more jars halfway through the project.

Two years later, this project is still one of the most fondly remembered participatory experiences at the museum--by visitors and staff. There was something special going on in that gallery. What was it?

THE RESEARCH

The challenge, of course, was to figure out how to evaluate the experience in a way that would help us identify the power of the project. We wanted to know whether the project truly had emotional resonance, and if so, how we could identify and document that.

Anna Greco did research in three ways: through in-person interviews with participants, surveys with participants, and observational analysis of the jars themselves. These methods revealed that the majority of participants had a meaningful experience with the memory jars that stuck--even in followup a year after the initial project.

I think most of us are familiar with interview and survey-based evaluation methods. I want to instead highlight the work Anna did to analyze the content of the jars observationally, which got at the question of emotional resonance in a more quantitative way.

Anna did two types of quantitative evaluation of the jars:

• she analyzed the jar contents, looking at how full the jars were. This was used as a proxy for time and creative energy spent on the creation of jars.
• she analyzed the text on the jar labels for length, for emotional content, and for intimacy. This was used to evaluate the amount of emotional energy dedicated to the jar activity.

In each of these analyses, Anna created a coding scheme to categorize the jars.

For the fullness of the jars, Anna created a seven-point scale, going from empty to full to full+ (additional decoration or objects on the outside of the jar) by percentage of jar full. It was fairly simple to identify whether a jar was 1/4 full or 3/4 full or had stuff popping out the top of it. The result here was surprisingly linear, with more than half of the jars full or full+. People used the stuff and they used it to the fullness of their ability. This could also be an argument for larger jars if we repeat the project.

For the content of the labels, Anna used three strategies.

• She counted the number of words in each label. This was an easy (if time-consuming) proxy for engagement. The average label had 17 words, though the maximum was 105. Again, this indicates a high level of engagement, especially given the size of the shipping labels provided.
• She created a numeric scale for the "intimacy" of each label. This was created with the help of Dr. Lauren Shapiro, a psychologist and former museum intern. Lauren and Anna created a scale of one to five where each level had specific elements to indicate intensity of the story shared, using signifiers like specificity of a memory, places, names, direct quotes, or medical information. 70% of the labels were a 3, 4, or 5, with only 4% at a 1. People got intimate, sharing intense stories of loss, special moments, and potent memories.
• She created a coding schema for "emotions" expressed in each label. Lauren helped Anna create a manual for language cues to signify any of nine emotions: happiness, love, gratitude/awe, sadness, pride, anger, fear, confusion, and mixed. 36% of the labels were happy, closely followed by 32% that demonstrated no clear emotion. Labels in the "no emotion" were "reporting" memories without explicit emotional language, as in "I remember going to the beach with my friends." Among the remainder, love, mixed emotions, gratitude, and sadness were the most frequent.

Intimate? Emotional?
Coding is useful but complicated
when the goal is to capture feeling.

We learned a few things from this process:

• Creating a coding scheme for text analysis is complicated, but it's worth it. Especially with such a large number of jars, it was really valuable to be able to distill the diversity of stories via a few key axes of intimacy and emotion. Unsurprisingly, developing the coding schemes to be able to be applied fairly consistently by people without a lot of specific training led to imperfections. But just going through the process helped us understand how we COULD quantify this kind of extremely qualitative content. You can check out Anna's coding scheme manuals on pages 67-69 of her thesis.
• Intimacy was the most useful indicator for this project, but still really complicated to measure objectively. If we were doing this again and had time to either code by emotion or intimacy but not both, I would choose intimacy. The intimacy measure was the clearest signifier of how people were using the Memory Jars and what stories they chose to tell. That said, we still had plenty of debate about what qualified as more or less intimate as the schema was being developed. Mention AIDS with no context, and you shot up to a 5 on intimacy. But tell an incredibly detailed story about your childhood, and you were likely to end up somewhere like a 3. When Anna did followup surveys, it became clear that many stories were more intimate to individual jar-makers than their coded labels reflected. It's arguable that the much simpler measure of word count is sufficient as a proxy; high intimacy stories had an average word count of 38, compared to the overall average of 17. If you use more words, you are probably going deeper into your story, which typically involves more commitment, intensity, and maybe--intimacy.   
• There are quantitative ways to measure creative participation. We keep trying to find ways to assign numbers to different kinds of participatory projects at the MAH. None of them are perfect, but all of them are useful in moving towards better design and better yardsticks for our work. The Memory Jar project allowed us to experiment with a more robust approach to quantitative evaluation of participation, and it got us interested in other ways to do it in simpler projects. For example, check out this quantitative method we use for comment boards. 
• Evaluating participatory experiences exposes new questions we actually care about, which are often different from what we thought we wanted to know. In trying to get a grasp of the kind of emotional resonance of the Memory Jars, we started having some interesting discussions about design and the end goals for our work. Would it be "better" if all the jars were a 5 on intimacy? If there were as many sad jars as happy ones? Just having this data opened up new ways of talking about what we are trying to achieve with our work. One of the goals we've stumbled into regarding participatory projects is the diversity of content presented. For some projects, ones that are designed to invite everyone to engage, we want to make sure the project absorbs a diversity of content in terms of language, emotion, intensity/intimacy, and creativity, so that every visitor can find their place in the project. In other projects, we actually want to "gate" the experience so that only people who are willing to devote X amount of time or intention will complete the project. Or we want to make it as simple and breezy as possible. Looking at the mix of what happened with the Memory Jars got us thinking about the ways we design for different kinds of outcomes, when we want diverse outcomes and when we want more focused sets.

What approaches have you used or considered for evaluating participatory projects? Please share your stories in the comments.

Building a Pipeline to the Arts, World Cup Style

It's World Cup time. And for the first time in my adult life as an American, that seems significant. People at work with the games running in the background on their computers. Conversations about the tournament on the street. Constant radio coverage.

If you are reading this outside the United States, this sounds ridiculously basic. Football/soccer is the world's sport. But in the US, it has only recently become something worth watching. For most of my life in America, pro soccer was considered something risible and vaguely deviant, like picking your nose in public.

But now it's everywhere. It's exciting. And it's got me thinking about how we build energy and audience for the arts in this country.

Barry Hessenius recently wrote a blog post questioning the theory that more art into the school day will increase and bolster future adult audiences for art experiences. Like Barry, I feel that more art in schools is always better. I also share Barry's skepticism that there is a direct, premier line between art in schools and adult audiences for art.

I got into museum work because learning happens in many places and many ways. School is just a small part of it. When we talk about what kinds of experiences create pathways to more arts participation as adults, we have to look in AND beyond school. We have to look at what kids are doing after school. What kinds of tools and media they can access. Who their role models are.

Consider soccer. Soccer did not transition into a national phenomenon in the US based solely (or even mostly) on school participation. Soccer isn't just part of school; it's part of life. Millions of youth play in afterschool and weekend leagues. International stars like Pele captured attention on the airwaves. Immigrant families found local leagues where they could participate and feel connected socially. Stronger youth leagues led to stronger college teams led to stronger Olympic and professional performance. And all of that led to more audience--at all levels of the game.

There have been pro soccer leagues in the US on and off for almost a century. But over the past fifty years, soccer has ascended on more levels to a stronger footing overall. Instead of just creating a league for top play and hoping an audience would show up, soccer is developing a constantly-refreshing audience by creating opportunities for kids as young as four to participate in AYSO youth leagues. AYSO goes out of its way to include kids with different abilities, with "everyone plays" as one of its chief tenets. At the same time as it promotes inclusive participation, AYSO is affiliated with the United States Soccer Federation (the pros). AYSO builds both the players and the audiences for soccer's future in this county at all levels.

This isn't just a soccer phenomenon. Across sports, there is a deliberate embrace of practice at all levels to strengthen participation. In basketball, there is street ball and varsity and college play and D-league and the NBA. All of these levels and types of play support a pipeline that accommodates players and audiences at different levels of skill, financial capacity, and enthusiasm.

This pipeline absorbs diversity in quality of play without "dumbing down" the experience or lessening the desire to achieve top levels of excellence. When we talk about embracing participatory experiences at non-professional levels, arts professionals often get worried that the best work will get drowned out in mediocrity. But the diverse world of sports has done an extremely good job of preserving the top levels of play. At the same time, it has cultivated active spectators who understand the subtle differences between the tiers. Imagine if people were as knowledgeable about the specific talents of top symphonic conductors as they are of top athletes.

What would that take? Arts already offer opportunities for participation, grand narrative, and diverse tiers of quality that are needed to make this pipeline a reality. We just have to connect the dots. Imagine if the world's leading symphonies were affiliated with the largest organizations for first-time instrument players. Imagine if every community theater was a development theater for a bigger one. If ballet theaters and ballet schools and dance in the park events were coordinated. If kids were checking out art supplies from the library. If maker fairs were connected to science centers.

To me, this kind of pipeline is necessary to build future audiences and practitioners in the arts. It just requires a few things:

• Mutual respect, coordination, and collaboration among organizations that work at different levels of expertise, budget, and scale. Instead of seeing differences in quality as competitive or embarrassing, we could see different tiers as part of a healthy ecosystem that builds rather than detracts. Enough art already exists to get this going--it just isn't usefully connected yet.
• Emphasis on developing both practitioners and audiences. Why can professional sports leagues charge hundreds of dollars for a ticket? Because they have built demand through hundreds of hours spent playing and watching the game for free or low cost. Sensible tiers lead to sensible discernment when it comes to paying for audience experiences.
• Diversity of opportunities to engage. School is only one of many venues where cultural experiences can happen. If we embrace the abundant diversity of venues and formats rather that declaring some "in" and others "out," we can spend more time building a field and less time parsing out who belongs there.

I dream of a time when we have as much attention and access to creative and cultural activities as we do to athletics. Sure, there will always be non-participants. There will always be audience gaps. There will always be top performers on the bench. But hopefully, there will be a culture of diverse people participating, cheering, and showing up.

If we can do it for soccer, I know we can do it for art.

Guest Post by Nora Grant: Lessons from A Year of Pop Up Museums

This post was written by my colleague Nora Grant, Community Programs Coordinator at the Santa Cruz Museum of Art & History. 

“Pop Up” has become an international buzz term to describe ephemeral, experimental projects--from pop up restaurants to pop up boutiques--but a “Pop Up Museum” is still somewhat mystifying. How can you take something as substantial and precious as a museum and add a pop up twist?

There are many different models, including The Pop-Up Museum of Queer History, The New York Met, SF Mobile Museum, and even a Pop Up Prison Museum.

At the Santa Cruz Museum of Art & History (MAH), we have been experimenting with a kind of pop up museum that is primarily created by the people who show up to participate. We’ve been popping up around Santa Cruz County for a little over a year and have had over 30 different pop up museums. Our primary goal for pop up museums is to bring people together in conversation through stories, art, history, and objects. Building off of Michelle DelCarlo’s pop up museum model, MAH pop up museums bring different people, perspectives, and projects to one central gathering place, enabling a democratic type of public curation.

What does this look like? Imagine a potluck in which instead of a dish, everyone brings an object and/or story to share with others. We choose themes and venues in collaboration with a community partner. People are invited to bring something on topic to share. When people show up, they write a label for their object and leave it on display. The museum lasts for a few hours on one day, with people coming and going as they please.

We favor this “potluck” approach because it:

• Empowers people to share meaningful stories and objects with one another 
• Enables the museum to step outside physical confines, and collaborate with community partners who wouldn’t ordinarily come to museum programs 
• Opens up conversation as to what it means to be a museum and who can participate in making one 
• Allows us to experiment with themes, content, and collaborations in an intimate yet short-lived, simple way 

In addition to pop up museums we facilitate locally in Santa Cruz, we also want to provide global support for anyone interested in having a pop up museum. We have created a free and downloadable pop up museum organizer’s kit. Check it out if you’re curious about choosing a strong theme, working with a collaborator, designing a portable structure, or tips for implementation.

Designing a pop up museum structure that is replicable by amateurs in diverse venues, appealing enough to attract a variety of participants, and portable is not easy. We continuously iterated our pop up museum format with different set up designs and language to realize a structure that satisfied these objectives. Having a lot of pop up museums and observing what did and didn’t work enabled us to learn more about our community while providing practical, real-life content for the organizer’s kit. 

While the kit offers a step-by-step guide for organizing pop ups, I want to share some of the more unexpected takeaways I’ve learned through this process.

Pop Up Outside the Museum 

It’s appealing to plan Pop Up Museums in conjunction with exhibits or museum events, but people are rightfully confused about a pop up museum taking place inside a museum. Like a café inside a restaurant, a museum inside a museum feels redundant rather than complementary. When framed by a larger museum, the pop up museum doesn’t embody as much individual vibrancy as it can’t be decontextualized from preexisting notions of said museum institution. When popping up in non-traditional exhibitory spaces, you cannot only prompt unexpected conversation, but also unite location and theme. For example, we had a pop up at an arboretum on Growth.

A Little Frame Goes A Long Way 

People like picture frames. They make the ordinary look special. They’re eye-catching. When we first started having pop up museums, we displayed objects on tables with black tablecloths, which had a simple yet flat aesthetic. So I bought vintage frames from a nearby thrift store, getting different shapes and sizes to accommodate various objects. The frames not only enhanced the aesthetic, but also visually communicated that the pop up museum serves as an open framework for the participant’s narratives. An empty frame is much more inviting than an empty table. Suddenly, people could pick and choose which frame they wanted, and design their own display within the communal show. Furthermore, open frames enable participants to physically touch exhibited content while demonstrating that the object still deserves special recognition.

Mix and Match Museum and Community Content 

One of the reasons we started the pop up museum was to challenge the idea that museums have an omnipresent authority over what is and what’s not “valuable.” We were surprised to learn that the pop up museum is actually most compelling when we exhibit objects from the museum’s collection alongside individuals’ objects. This bridges institutional and community-created content. By sharing the same space, you’re illustrating how a personal object can have just as much story value as a museum object. This mixing and matching ties into another conversation around what a “museum” means to people. People certainly have diverse views and relationships to museums, but I found that most of our collaborators were excited to partner with a museum because it validated their project or object. Something about the idea of a museum carries a lot of weight, and the Santa Cruz community has responded well to pop up museum collaborations. This is not to say that everyone loves the MAH or that everyone sees our pop up museum as museums for that matter. But we did notice that participants and collaborators were more attracted to having a pop up museum in partnership with the MAH rather than throwing one on their own.

Serial Pop Up Museums Sustain Interest 

Pop Up Museums are ephemeral, and because of their brevity, it can be difficult to maintain and sustain momentum for each one. We experimented with serial pop up museums and held six pop up museums thematically tied to an exhibition last fall. Held in collaboration with six different partners, taking place in six different locations, and occurring on consecutive Saturdays, each pop up expanded the overarching theme of the exhibit. Unlike other pop up museums which live for one day, serial pop up museums have the advantage of reaching a larger audience and encouraging repeat participation. 


What’s wonderful about the pop up museum is that it’s a flexible format for sharing. The changeability is part of its charm. We don’t know exactly how, or where, or with whom it will pop up. But like the blank labels or empty frames we leave out on the table, pop up museums will continue to invite and support public conversation, personal empowerment, and open-ended narratives.

Disinterested or Excluded? Two Very Different Forms of Non-Participation

"I hate museums."

Imagine two people saying this.

For one of them, the statement is grounded in disinterest. She's visited museums, she didn't enjoy them, she's more of a sports person, it's not part of her life, it's not her thing.

For the other, the statement is grounded in exclusion. He didn't feel welcome in museums, feels like they aren't for him, he wasn't invited, it isn't comfortable.

How can you tell the difference between these two people?

It's almost impossible to do so. They are likely to use similar language--language of disinterest--in describing what turns them off. It's safer to say "I'm not into it" than "I feel excluded." Sometimes the difference isn't even apparent to the person speaking. People can mask exclusion as disinterest unconsciously as a protective measure against what they are really experiencing.

From the institutional standpoint, this difference matters. It matters when we think about who our organizations are for and what we are willing to do to invite those people to participate. It may be ridiculous for an institution to be "for everyone." But how do we decide who it's ok to disregard?

I think it's ok to strategically disregard people who have disinterest in the content or format of your work. It's not ok to ignore people who have interest but feel excluded.

It's not only politically problematic to ignore them--it's also a bad business practice. Excluded yet interested people are potential participants who can and will be engaged if the organization is more open to them.

I fear that too often, professionals mis-identify exclusion as disinterest. It's easy to do. We mirror what we hear from non-participants about "not liking it." We stick with those who like it and feel included. We perpetuate the exclusion. And then we wonder why some people don't show up.

What's the alternative? We can probe deeper. We can start looking for signs of interest and building on them. We can start identifying the code words that turn "it's not for me" into "I choose to spend my time elsewhere" versus "you made me feel unwelcome." We can be ok with disinterest. We can work on inclusion. We can make a change where it will matter.