Evaluating Advocacy: A Model for Evaluating Public Deliberation Efforts

The Harvard Family Research Project publishes The Evaluation Exchange, a periodical on emerging strategies in evaluation. The Spring 2007 issue (volume XIII Number 1) is the first of several issues they will publish covering “Hard to Measure” evaluations. Their first issue dealing with evaluating advocacy efforts has several ideas relevant to evaluating public engagement in public policy. (Future issues focusing on community organizing and participatory democracy promise to be even more relevant to our work.)

In her introduction to this issue, Julia Coffman discusses how evaluation of advocacy differs from evaluation of other programs and services. The differences that she cites applies to the work that we do in engaging people in deliberating public policy issues.

Advocacy strategy typically evolves over time
Certainly deliberation as a strategy for engaging the public in working through complex and divisive issues has changed during the seventeen years I have been involved in this work. In my early days of doing this work, people’s eyes would glaze over when I invited them to participate in a forum. (My first forum, Growing up at Risk was a notable exception attracting an overflowing roomful of participants and the evening newsreporter, much to the alarm of this “then wet behind the ears moderator.”) Over the past five years, I’ve noticed that I can barely get through my second sentence before people interrupt me with, “That’s exactly what we need in this country – a better way to talk about these issues!”

Activities and desired outcomes can shift quickly
I see this happening with public deliberation in three ways. People are increasingly framing their own issues locally, there is a greater emphasis within the NIF network for forums to lead to action, and forums are no longer stand-alone events, but part of a larger strategy for solving problems. One shift I am introducing to the work of Texas Forums is to embed deliberation into the everyday thinking and work of nonprofit organizations and communities. For example, we are partnering with E3 Alliance and Austin Voices for Education and Youth to conduct forums on the Achievement Gap in seven communities in Austin. Rather than “ride into town and hold a forum” Austin Voices will work closely with each community to develop a planning team that will provide the infrastructure for future community engagement efforts of E3 on other aspects related to education.

The policy process itself is unique
We have much to learn in this area. I fear that we have not done a good job of telling the story of what people want policy-makers to think about and how we want them to think. Just last evening, As a result, they are often stymied in their understanding of what the public will support because they are stuck with unstable opinion polls. I saw this last night in the public forum I attended in my little village. People panicked when the mayor suggested that properties in the Extraterritorial Jurisdiction (part of the Village, but not in the village limits) could be transferred to an encroaching city “if it made sense.” During the Q&A, I suggested that perhaps the mayor and aldermen could allay our fears if they could describe the criteria they use when deciding what “makes sense.” In other words, I wanted to know what their priorities are…what motives and values would drive their decision-making. Although I think I eloquently phrased my question (did you expect me to say otherwise?) they were unable to answer it beyond, “There’s no way we can tell you what decision we will make in every circumstance.” We need to consider the current policy process and develop better ways of communicating the results of our forums. (For a start, we are offering an online workshop on “Reporting on Your Forums” on April 5.)

Most advocacy organizations are small in terms of their size and their capacity to manage evaluation
Much of the dialogue and deliberation work across the country is being driven by volunteers, people within organizations who also have other responsibilities, or small nonprofits.

I would also note that a lack of transparency in public policy and the nightmarish task of deconstructing the federal budget are two other challenges to advocacy evaluation. Bill Bradley, in his “just released today” book, The New American Story calls for greater transparency in the federal budget. He (how could I not agree) calls for the budget to be on the Internet with keyword accessibility so that users could easily navigate the quagmire to find how money is being spent for various purposes. The impact of advocacy efforts would become even more easily identified if you take it a step further and include the technology that enables a user to receive e-mail alerts whenever there are changes related to their key issues. We aren’t there, yet, but there are efforts coming together around that very issue. (But I digress!)

The rest of The Evaluation Exchange is devoted to evaluations to watch, best practices and expert advice. Good stuff here for airplane reading! I’ll add comments here as I learn new ways for us to apply advocacy evaluation to our own work.

Technorati Tags: Bill Bradley, Evaluation, Harvard Family Research Project, Texas Forums, The New American Story