Archive for the ‘Evaluation’ Category

NOTE: For more information about this project, check out the following:

Earlier this week I participated in a Kettering symposium at the National Press Club to discuss the findings of their recent report, Helping Students Succeed: Communities Confront the Achievement Gap. A prestigious panel with students, a parent, school administrators and teachers, researchers and a mayor were on hand to share their response to this report that documented what happened in ten communities around the country when people came together to deliberate what they could do to help close the Achievement Gaps. Our own Dr. Patty Shafer, San Marcos School Superintendent was on hand to talk about the impact that the community dialogues have had on how people in her community now work together to meet the needs of an increasingly diverse population.

Over the years I have written in this blog about this project in Central Texas and Texas Forums’ fruitful partnership with E3 Alliance and the many school districts and community members that have pledged time, energy and resources to work together on education with an eye toward regional solutions. My next step in this venture will be to work with the Blueprint Team for Goal 4: Central Texas as a community prepares children to succeed. Stay tuned for more postings about community engagement and evaluating progress toward this hard to measure goal.

But back to the DC symposium…

I was proud to represent Central Texas at this symposium and even more proud that our work is prominently featured in a forthcoming documentary about this project. We had a chance to preview the trailer for this documentary at a working luncheon following the symposium.

During the working luncheon following the symposium, I was charged to lead a discussion about the Achievement Gaps and to take notes for the Kettering Foundation. Below are snippets from our conversation.

What in your judgment is the most significant finding from the study? Did anything surprise you?

Several of the participants were surprised that so few people were familiar with the achievement gap issue. One participant wondered what this meant for the way the issue was named and framed. In other words, perhaps the public has a different perspective on what is really at issue when it comes to disparities in educational accomplishment. The project researcher noted that, based on his experience, those within the system hesitate to raise the issue of achievement gaps even to the extent of presenting data about the gaps in unfriendly, inaccessible formats so that the School Board would not readily detect the seriousness of the issue. Our superintendent countered that her annual performance review is heavily dependent upon being able to demonstrate strides toward closing the gaps. While many districts may be less than transparent about the data and may be reluctant to confront the issue, the ability to hide the seriousness of the issue may vary according to state reporting requirements. However, it is still clear that there is a huge disparity in the student achievement across the country as well as disparity in how much we know. One participant asked, “who is looking at this closest?”

One lunch guest commended the forum participants for their insight and willingness to confront the complexity of the issue and not grab at easy fixes. Instead, the forum participants rejected these easy fixes and moved beyond their pet cause or, as one participant described it, they gave up being “one trick pony advocates” and opened up to a range of possible actions. We briefly discussed how this shift occurred and concurred that the structure helps people bypass their firm notions about what should be done and makes it comfortable for them to entertain other options. The structure also creates the space for parents who have never been asked to realize “we can be part of the solution.” One participant cautioned that we must be diligent about the language that we use and aware of how language can keep people out of seeing themselves as part of the solution.

Since one of our participants had traveled to several sites to interview forums participants in depth for a documentary she is producing, we asked her to compare how different communities were defining he cause of the achievement gaps. In Central Texas, the primary driver (San Marcos community in particular) was the changing demographic and rapid growth of the ELL population.

She reported that poverty was also an issue in San Marcos, but not in the same widespread way it was expressed in Helena where poverty seems to be fueling a sense of hopelessness. The hopelessness is exacerbated by the concern that an improvement in the school and in the outcomes for children would lead to youth flight and the demise of the community. And yet, the community seems stymied from making the kind of improvements (renovating old buildings, attracting new business) that would be necessary to attract employers that could provide stimulating economic opportunities for new graduates. As one Helena participant noted in the documentary trailing, “the running joke is, ‘the last person out of Helena, turn out the light.’”

The gaps in Bridgeport are caused not just by poverty, but by the allocation of resources. While many in the community are poor, participants identified the disparity in resources between schools as an important consideration. Students feel safe while at school, but they don’t have the same level of security in their communities.

Because we were fortunate to have three young scholars in our group, I asked them how these conversations about the achievement gaps relate to their recent (more so than the rest of us at the table) high school experiences.

Astonishingly, one young woman raised in New Mexico responded, “It would have been great if someone had cared about this at my school.” She then relayed a story about being bused to a school south of town where there was a “mish-mash” of kids, what could have been a rewarding multi-cultural learning experience, but was really a holding place for kids who were given no direction, incentives, or experiences. It was the opposite of what Dr. Edmund Gordon (a panelist at the National Press Club) described as a community dedicated to education that included a school. Instead, the picture she painted was of a walled-in school in the middle of rich cultural opportunities that the students never experienced. [Ed. Note: Dr. Gordon is a brilliant thinker with remarkably diverse and deep scholarship and I felt blessed to be in attendance for his comments. Check out his biography and this brief video tribute from EdLab – Teachers College, Columbia University to him for more information.] Of her school of approximately 400 students, only100 students graduated (admittedly some moved away, and a meager 5 ventured out of state to pursue higher education opportunities. Other participants noted that even in the shadow of our nation’s capitol, there are entire school districts whose students have never visited the Smithsonian.

The experience of a young woman who went to high school in Denver was equally as dismal. She described multiple gaps – the number who did or did not complete high school, those who held high grade marks vs. those who did not, and those who pursued academic studies. She remembers having these conversations as she speculated on what it meant and why it occurred that she was only one out of two Black students in the Advanced Placement track at her high school, but she doesn’t remember that any of these conversations were intended to uncover the reason nor did they result in any changes.

A third young scholar did not remember any conversations about this topic in her school and was impressed to learn that students in the forum wanted adults to have high expectations of them. She reflected on how the naming of the problem indicates who will be involved in “fixing” the problem. The issues of education and youth seem so personal and so local that there is a danger of losing this when the issue becomes national or global.

Returning to the earlier challenges in Helena one participant summarized the tension facing communities, particularly the rural and declining communities: How do you preserve a local way of life while exposing kids to the world beyond? Some piece of the solution may be found in examining communities like York PA which has become a multi-ethnic community where kids grow up and go to college, but after they graduate, marry and start a family, many of them return to York because it offers a way of life that they want to give to their children.

Ultimately, our table agreed that education must be more broadly defined as an ongoing activity that occurs in lots of places, not just in school and that the over-riding question we must seek to answer is:

How can we make sure that
every student has every opportunity to succeed that they need?

Read Full Post »

I know that numbers alone do not always measure impact and in this case none of the numbers really mean a hill of beans. (I am really put off by those twitterers who engage in contests to see who can get more people to follow them, for example.) Still, it’s fun to look at the numbers every so often. Here are a few from Texas Forums:

181 – number of people following LBJnow twitter (not huge, but still…)
1369 – number of updates LBJnow has twittered
312 – number of posts to the Texas Forums Blog since October 2006
1,451 – largest number of views of TF blog in one month
1.097 – number of Texas Forums photos online at Flickr
62 – number of videos uploaded to our site at youtube

Read Full Post »

One of the objectives of the Central Texas D&D Summit held on April 19 at the LBJ Library was to:

a) Identify specific local D&D efforts that could be used as examples of the National Coalition for Dialogue and Deliberation’s (NCDD) ‘Seven Challenges Facing the D&D Community’;
b) Members of the NCDD_CenTX team (to be determined) will then document these examples (according to a format yet to be determined) for presentation at the conference.

While we made great headway toward our other objectives to build a Central Texas Network, we didn’t specifically tie our practices and learning back to the seven challenges. NCDD and Civic Evolution have been hosting an online dialogue about these seven challenges, but that doesn’t satisfy our objective to draw on the expertise of the Central Texas region.

If you are a Central Texas D&D practitioner or scholar and have struggled with these issues, please let us know what you are learning and how we can pool our resources to tackle these challenges.

Here are the challenges:

Bringing D&D skills and perspectives into mainstream society and institutions

Challenge A: Embedding D&D in Systems

Embedding D&D in systems (governance, schools, organizations, etc.) – as opposed to just putting our energy into isolated D&D events and programs.

Challenge B: Framing this Work in an Accessible Way

Articulating the importance of this work to those beyond our immediate community (making D&D compelling to people of all income levels, education levels, and political perspectives, etc.) – and helping equip members of the D&D community to talk about this work in an accessible, effective way.

Challenge C: Proving This Stuff Works

Proving to power-holders (public officials, funders, CEOs) that D&D really does work, and creating/propagating quality evaluation tools for practitioners to use that can feed into research. In the private sector, demonstrating how D&D contributes to the bottom line.

Challenge D: D&D to Action and Policy Change

Strengthening the link between D&D and community action and policy change.

Strengthening the D&D community

Challenge E: Walking Our Talk

Addressing issues of oppression and bias within the D&D community.

Challenge F: Regional D&D Networks

Fostering the development of regional D&D networks and gatherings.

Challenge G: International Connections

Finding ways to readily learn from what D&D innovators outside of the U.S. are doing.

So what have we learned about these important challenges?

Read Full Post »

The participants in the small groups at the Central Texas D&D Summit had almost forty-five minutes to share the lessons and insights they have gained during the course of their dialogue and deliberation work.

Steven Fearing set up the groups with the following comments.

  • This meeting is not a workshop on dialogue and deliberation techniques. Instead it is an opportunity for us to get to know each other and to document what we have done and learned. More importantly, we want to build relationships and find out how we can work together in the future.
  • You will be working in small groups at five tables
  • Each table has a facilitator
  • You have a template prepared by Sunni where you can capture in words and pictures from your group’s dialogue.
  • You will have 45 minutes for this portion and should feel free to take breaks as needed
  • Goal for our time in these small groups is to share and capture key learning, insights, and challenges related to dialogue and deliberation in community work.
  • After 45 minutes, we will come together to capture common themes – such as assets and resources, challenges and opportunities. Sunni will help us integrate all of the ideas we discuss in our groups and chart on our templates into a graphic narrative.

Each group had a beautiful graphic map drawn by Sunni Brown where they could capture their insights. The groups were lightly facilitated by Rod Reyna, Susan Schultz, Tobin Quereau, and Mary Thompson. The facilitators charged the participants:

  • Think of a time when you brought people together to work on an issue or community problem. What lessons have you learned about using dialogue and deliberation for helping people work together? These may be lessons you learned from your successes or things you learned that you would do differently.
  • Think also about challenges you have faced and what you would like to do better, and
  • What else do you need to be more effective in using dialogue and deliberation in your work with communities?

Here are the results of their small group dialogue:

Yellow Group

yellow group

Susan Schultz, Neil Meili, Stephanie Nestlerode, Steve Swanson, Lindsay LeBlanc

Blue Group


Mary Thompson, Ed Sharpe, Margaret Valenti, Robyn Emerson, Oliver Markley, Patricia Wilson

Red Group


Tobin Quereau, Ann Brudno, Jenny Meigs, Tom Moran, Landon Shultz, Mike Aaron, Leilani Rose

Green Group


Rod Reyna, Sherry Lowry, Robena Jackson, Cathey Capers, Juli Fellows, Steven Fearing

As the participants described their templates, Sunni captured their themes:


Prior to the reporting out and reflection, Erin Kreeger and Taylor Willingham had a charge for the group:

As you listen to the groups’ posting their templates, listen for assets and opportunities. Think about the opportunities that you identified in this room that will help you in your work. Perhaps you identified asset or opportunities that involve:

  • Connecting with someone else in this room or someone who needs to be part of this community
  • Participation in an event or activity
  • Contributing your expertise or resources

Make note of the ideas as they come to you. After every group describes their template we will have time for collective reflection that Sunni will capture for us in graphic form.

Here are some of the opportunities the group identified to connect:


In keeping with our spirit of reflection and “continuous improvement” (that term is here for Charles’ benefit!) Charles Knickerbocker led us in a period of reflection on the meeting. He asked them what worked and what did we need more of. Here’s what the group had to say:

what worked

Read Full Post »

[This report is being filed by Erin Kreeger, a member of Texas Forums, graduate of the Fielding Graduate University’s Certification in Dialogue, Deliberation and Public Engagement and an adviser to the University of Houston Downtown’s Center for Public Deliberation. Erin will be an ongoing guest blogger for Texas Forums so check back often to hear her insights!]

On April 4th and 5th around 25 incredible people gathered at The University of Houston – Downtown Center for Public Deliberation for a powerful workshop on moderating and recording public deliberation forums. These forums are opportunities for people to join together with others to talk about difficult issues, gain new insights on ways to approach those issues and to choose ways to work towards creating powerful individual and group action, including influencing public policy. The workshop provided an opportunity for people who may not have done something like this before to learn from some seasoned experts, to learn from each other, to practice participating in two deliberative forums (one on the achievement gap in education and one of the energy problem), to moderate a forum, to record insights and themes from the forums and to begin building a community of practice. How great is that!

Though two day workshops can be challenging to design in a way that’s flexible enough to adapt to people’s needs and questions yet structured enough to end on time, this planning team did that brilliantly – keeping us engaged for the entire 2 days – including 7 hours of Saturday time. Here’s what participants had to say about what worked really well and what could be done differently next time.

What I’m taking with me/Keep It!

  • Role playing/Practice moderating forums
  • Intentional prep activities – not arbitrary
  • I was engaged
  • Power of communication
  • The workshop kept moving
  • Good to have to jump into activities
  • Having multiple instructors
  • The printed materials to read later instead of being read to
  • Applicable – can apply ideas right away
  • Great modeling of practices
  • Food
  • Strength of moderators and their stories

What I’m leaving behind/Drop it

  • Need clearer directions to get to the center
  • More vegetarian food options/easy to identify veggie food
  • More signs in building directing to room
  • Want video of the practice forum

At the end of the workshop, one participant said that she felt she had found her public deliberation family.  I find that feeling of community is inspiring and happens a lot in this line of work.  But what’s especially exciting to me about this particular workshop is that The University of Houston Downtown Center for Public Deliberation in partnership with Texas Forums has the skill, desire and dedication to provide those family member with the resources they need to stay connected and to convene, moderate and record public deliberation forums so that community members of all backgrounds have the opportunity to meet with each other in a public dialogue, to identify the concerns they hold in common and to create action on issues that are important to them.  That’s something I’m excited to be a part of.  It’s a great example of inviting change.

Read Full Post »

We have not yet begun our forums on Too Many Children Left Behind: Closing the Achievement Gaps in Central Texas, but we already have some success stories to report.

Here is a report filed by Susan Dawson, E3 Alliance.

Each community started off with a Too Many Children Left Behind launch team of (at least) a district superintendent, a higher ed leader, a local politician, and a business leader.  This launch team was asked to own and commit to helping with the final plans coming out of the dialogues, help find resources needed to hold the dialogues, and to identify a diverse group of 8-10 “grass tops” Community Champions to roll up their sleeves and plan/manage the dialogues process in their local communities.  The San Marcos Launch Team broke down their community into different groups and identified  over 60 people that they wanted to be on the Community Champions team!  Then that Champions team, the first time they met, were excited about immediately launching into discussing the implications of achievement gaps on their community, and had to pull themselves back into planning mode.

In one community, some of the Champion team wanted to be moderators, but they also wanted to be able to participate actively in the dialogues about their schools.  In the end, they decided to volunteer to moderate in another community so they could do both!

One person at the first moderator training session was an AP student and the head of student government at her local high school.  After participating in training, she was so excited about engaging students of different backgrounds that she took the discussion guide back to her school and immediately started to launch her own Too Many Children Left Behind dialogues with students at her school.

We sent an invitation to attend one of the Too Many Children Left Behind community dialogues to all the superintendents from across the region.  One superintendent volunteered to be a moderator at another community’s dialogues!

Read Full Post »

It would be interesting if they said something we hadn’t heard. Barack makes a joke about “No Child Left Behind” and talks about the money left behind but doesn’t get a big response here. Bonnie observes that most of the people here are taking it all blandly and no one is getting up and cheering. That’s the plight of this kind of crowd.

And Dennis Kucinich can get cheers from the crowd at the forum but little response from these folks. Mike Gravel has the freedom to talk truth because nobody takes him seriously.

So let’s talk about this event, as Yours Unruly plans to jet, and what the overview should be: the crowd should have been larger certainly, there should have been more journalists here to get the opinions of the average person – most of whose concerns are really about the economy, health care and having a government that isn’t riddled with corruption. I heard that tonight, walking from one table to another. I watched the people sitting here, Black, Asian, Latino and White and most them – as mentioned – had no idea about the “Covenant” but had serious ideas about what they wanted for this country. Many of them expressed their desire for common ground. This Blogger was happy to hear that.

At the same time, as a long-time journalist, I would rather have attended an event where more of the people got to speak instead of watch, an event where there was involvement and interaction instead of observation. Passivity is not what is needed now. Action is what is needed now.

The next stage of politics is involving the polity.

This was certainly worth doing but more is needed, more conversation.

Read Full Post »

 On the evening of May 1, ten members of Texas Forums participated in a research project for the Kettering Foundation. They watched an unedited version of Public Voice, a documentary filmed earlier in the day at the National Press club featuring panelists who were commenting on videotaped excerpts of National Issues Forums on energy. The panelists participating in this documentary were:

Host/Moderator:  Frank Sesno

E. J. Dionne, Columnist, Washington Post
Betty Sue Flowers, Director, Lyndon Baines Johnson Library and Museum
Jay Hakes, Director, Jimmy Carter Library
Mike Johannes, U. S. Secretary of Agriculture
Senator Mary Landrieu, Louisiana
David Mathews, Kettering Foundation
Congressman Charles Gonzalez, Texas
Andrea Seabrook, Congressional Correspondent, NPR
Senator Jeff Sessions, Alabama
Jerry Taylor, CATO Institute
Roger Wilkins, Author and Analyst

This is the Kettering Foundation’s description of the research:

“A Public Voice brings together policymakers and policy elites on a panel to react to and discuss the implications for their own work of scenes from publics grappling with a significant national issue, this year, energy. As the publics deliberate, they identify why the issue is important to them, what things highly valued by them they see at stake in the issue, why they cannot get everything they want regarding the issue, and so, in the end, struggle with what trade-offs among things highly valued they may be willing to make, and thus, what kind of permissions for action they would open up for policymakers.

In addition to demonstrating these qualities of public deliberation through A Public Voice, Kettering’s research interest lies largely in seeking to understand the conditions under which policymakers and policy elites come to recognize the contributions that a deliberative public can make to their own work. In conducting this research with policymakers, we bear in mind five underlying questions, greater understanding of which may help us understand better how a deliberative public can more effectively relate to its elected representatives, and how our representative institutions may become more responsive to a thoughtful, deliberative public. “

Below is a brief report filed by Marla Crockett who facilitated the discussion on May 1. We will prepare a longer report and meet in D.C. on June 8 with colleagues from the Ford and Carter libraries and Saddleback Community College who also participated in this research.

“Citizens who watched A Public Voice at the LBJ Presidential Library in Austin saw and heard two different discussions Wednesday night. The seven women and three men were virtually unanimous in saying that edited comments from the public forums on energy were deliberative and held their attention, while experts on the panel were at times “dismissive,” and “spoke from a script.”  Members of our group praised a few panelists, including Carter Library Director Jay Hakes and Secretary of Agriculture Mike Johannes, for really listening to the public state the problem, weigh trade-offs and reflect on leadership.  However, the consensus in our group was that too many experts focused on positions and strategies instead of on interests and concerns.

What came through the loudest, however, was an unhappiness with the program’s format.  While recognizing that the demands of television might be at odds with a deeper conversation, our Austin panel wanted to see a dialogue between the experts and the public.  The exchanges led by moderator Frank Sesno were “jarring,” and too much like “Crossfire,” several people said.  One woman felt more optimistic at the end of the program, because it seemed as though public officials and journalists really understood the public’s desire for change.  But one of the men said that if citizens had been allowed to participate in the discussion, they would have held the politicians more accountable on issues like the influence of money on energy policy.”

Read Full Post »

The Harvard Family Research Project publishes The Evaluation Exchange, a periodical on emerging strategies in evaluation. The Spring 2007 issue (volume XIII Number 1) is the first of several issues they will publish covering “Hard to Measure” evaluations. Their first issue dealing with evaluating advocacy efforts has several ideas relevant to evaluating public engagement in public policy. (Future issues focusing on community organizing and participatory democracy promise to be even more relevant to our work.)

In her introduction to this issue, Julia Coffman discusses how evaluation of advocacy differs from evaluation of other programs and services. The differences that she cites applies to the work that we do in engaging people in deliberating public policy issues.

Advocacy strategy typically evolves over time
Certainly deliberation as a strategy for engaging the public in working through complex and divisive issues has changed during the seventeen years I have been involved in this work. In my early days of doing this work, people’s eyes would glaze over when I invited them to participate in a forum. (My first forum, Growing up at Risk was a notable exception attracting an overflowing roomful of participants and the evening newsreporter, much to the alarm of this “then wet behind the ears moderator.”) Over the past five years, I’ve noticed that I can barely get through my second sentence before people interrupt me with, “That’s exactly what we need in this country – a better way to talk about these issues!”

Activities and desired outcomes can shift quickly
I see this happening with public deliberation in three ways. People are increasingly framing their own issues locally, there is a greater emphasis within the NIF network for forums to lead to action, and forums are no longer stand-alone events, but part of a larger strategy for solving problems. One shift I am introducing to the work of Texas Forums is to embed deliberation into the everyday thinking and work of nonprofit organizations and communities. For example, we are partnering with E3 Alliance and Austin Voices for Education and Youth to conduct forums on the Achievement Gap in seven communities in Austin. Rather than “ride into town and hold a forum” Austin Voices will work closely with each community to develop a planning team that will provide the infrastructure for future community engagement efforts of E3 on other aspects related to education.

The policy process itself is unique
We have much to learn in this area. I fear that we have not done a good job of telling the story of what people want policy-makers to think about and how we want them to think. Just last evening, As a result, they are often stymied in their understanding of what the public will support because they are stuck with unstable opinion polls. I saw this last night in the public forum I attended in my little village. People panicked when the mayor suggested that properties in the Extraterritorial Jurisdiction (part of the Village, but not in the village limits) could be transferred to an encroaching city “if it made sense.” During the Q&A, I suggested that perhaps the mayor and aldermen could allay our fears if they could describe the criteria they use when deciding what “makes sense.” In other words, I wanted to know what their priorities are…what motives and values would drive their decision-making. Although I think I eloquently phrased my question (did you expect me to say otherwise?) they were unable to answer it beyond, “There’s no way we can tell you what decision we will make in every circumstance.” We need to consider the current policy process and develop better ways of communicating the results of our forums. (For a start, we are offering an online workshop on “Reporting on Your Forums” on April 5.)

Most advocacy organizations are small in terms of their size and their capacity to manage evaluation
Much of the dialogue and deliberation work across the country is being driven by volunteers, people within organizations who also have other responsibilities, or small nonprofits.

I would also note that a lack of transparency in public policy and the nightmarish task of deconstructing the federal budget are two other challenges to advocacy evaluation. Bill Bradley, in his “just released today” book, The New American Story calls for greater transparency in the federal budget. He (how could I not agree) calls for the budget to be on the Internet with keyword accessibility so that users could easily navigate the quagmire to find how money is being spent for various purposes. The impact of advocacy efforts would become even more easily identified if you take it a step further and include the technology that enables a user to receive e-mail alerts whenever there are changes related to their key issues. We aren’t there, yet, but there are efforts coming together around that very issue. (But I digress!)

The rest of The Evaluation Exchange is devoted to evaluations to watch, best practices and expert advice. Good stuff here for airplane reading! I’ll add comments here as I learn new ways for us to apply advocacy evaluation to our own work.

Technorati Tags: , , , ,

Read Full Post »