New Resources: New and Fascinating Resources

The tide that has swept experimental program evaluation to the forefront of knowledge building about social policy is suddenly ebbing. By Lisbeth B. Schorr Jan. 8, 2016

http://ssir.org/articles/entry/reconsidering_evidence_what_it_means_and_how_we_use_it

The latest winner of the Nobel Prize in Economics, Princeton’s Angus Deaton, was described by Justin Wolfers in the New York Times as “an influential counterweight against a popular strand of econometric practice arguing that if you want to know whether something works, you should just test it, preferably with a randomized control trial. In Mr. Deaton’s telling, the observation that a particular government intervention worked is no guarantee that it will work again, or in another context.”'

Vincent DeVita, MD, former head of the National Cancer Institute and physician-in-chief of the Memorial Sloan Kettering Cancer Center, is also skeptical, but in a medical context. In his book, The Death of Cancer, he characterized evidence-based guidelines for the treatment of cancer as “backwards looking.” He wrote, “With cancer, things change too rapidly for doctors to be able to rely on yesterday’s guidelines for long. Reliance on such standards inhibits doctors from trying something new.”

Evaluation guru Thomas Schwandt also urges caution in how we approach documenting effectiveness. In the 2015 book, Credible Evidence in Evaluation and Applied Research (S. Donaldson, C. Christie & M. Mark, Eds.), he wrote, “ ... the field of evaluation seems captivated by discussions of methods needed to pro­duce evidence of impact ... [distracting] us from care­fully attending to a variety of important issues related to evaluative evidence and its use.” He suggests that “the term evidence base must be interpreted with caution: To claim that evidence may figure importantly in our decisions is one thing; to claim it is foundation for our actions is another. We would be well advised to talk about evidence-informed decision making instead.”

From a philanthropic perspective, Vivian Tseng, vice president of the WT Grant Foundation, writes in a similar vein, in “Evidence at the Crossroads”: “A narrow focus on evidence-based programs encourages people to run after silver bullet solutions that are not necessarily aligned with the myriad other interventions that they are running.”

These are compelling points of view. When it comes to addressing serious problems such as poverty, and race- and income-based disparities in health and education, the world is beginning to discover that the most effective interventions consist of far more than individual, circumscribed programs. This may help to explain why the tide seems to be shifting away from a narrow focus on experimental evidence of program impact.

Thinking that we’re probably only in the early stages with this realization, I was surprised that in a session on this subject at last November’s American Evaluation Association meeting, the message that we need a broader approach to evidence was enthusiastically received. There seemed to be considerable agreement that a narrow focus on trying to identify which programs “work” is actually keeping us from getting better results and that the social sector’s program-centric focus has been based on several erroneous assumptions. These include that individual, stand-alone programs can achieve ambitious goals; that if we know from RCTs that a program works in one place, it will work everywhere; and that innovation won’t be discouraged by an over-arching reliance on programs that have been shown to work in the past.

No one questions the importance of evidence. But it is time for all of us to think more expansively about evidence as we strive to understand the world of today and to improve the world of tomorrow.

Don Berwick, health policy reformer extraordinaire (and my colleague in the Friends of Evidence), describes the situation this way: “The world we live in is a world of true complexity, strong social influences, tight dependence on local context—a world of uncertain predictions, a world less of proof than of navigation, less of final conclusions than of continual learning.” (“Eating Soup with a Fork,” Keynote, 2007 Forum on Quality Improvement in Health Care.)

To get better results in this complex world, we must be willing to shake the intuition that certainty should be our highest priority. We must draw on, generate, and apply a broader range of evidence to take account of at least five factors that we have largely neglected in the past:

1. The complexities of the most promising interventions
2. The practice-based evidence that spotlights the realities and subtleties of implementation that account for success
3. The importance of fitting interventions and strategies to the strengths, needs, resources and values of particular populations and localities
4. The heavy context-dependence of many of the most promising interventions
5. The systematic learning and documentation that could inform future action

One way to accomplish this goal is for all those involved in intentional social change— including philanthropies, public policy makers, and nonprofit organizations—to go about the business of knowledge development in a way that would enable us reliably to achieve greater results at scale in tomorrow’s world by making sure that all public and philanthropic funding is evidence-informed. For a start, this would require:

• Investment in structures that could identify the common underlying elements of diverse attempts to reach similar goals
• The development and maintenance of directories that would address the contextual factors, whether and under what circumstances programs are likely to be effective in new settings and populations, and add a focus on the work that focuses on systems and community change
• A means to identify ways of making systems more hospitable to interventions that are evolving and improving, and take seriously the challenges of implementation

This approach to knowledge development and learning, in the United States at least, would contribute substantially to the nation’s capacity to solve big problems. Of course, solving big problems takes political will, not just more and better knowledge. But by becoming smarter in how we approach the generation, analysis, and application of knowledge and evidence, we can contribute mightily to building the needed political will.

- See more here

Linda Silka Maine reinvents research to tackle ‘wicked problems’

We need stakeholder-engaged solutions-focused interdisciplinary work if our scarce science resources are to be mobilized to help solve wicked problems.”

Read more: https://outlook.office.com/owa/?path=/mail/search/attachmentlightbox

Frank Drummond, a researcher with the University of Maine, talks about a blueberry study in Jonesboro in 2011. By Linda Silka, Special to the BDN (BDN File)

Posted March 11, 2016, at 10:21 a.m.

Scientific research plays a big role in our lives. We have come to rely on it in so many ways. It seems that every day we hear someone say, “well, research has shown…”

But we are coming up against the limits of research as it traditionally has been done. The image of science — the one in popular culture of the lone lab-coated researcher hitting upon a brilliant idea — is fast becoming outdated.

Instead, efforts are being made to find better ways of ensuring that research helps solve our increasingly tough societal challenges. Maine is leading the way in developing some of these new forms of science.

Consider several of our problems: poverty, pollution, failing school systems, racism and discrimination, income inequality, elder abuse. Pick up the daily paper and one is beset with story after story about these seemingly overwhelming problems. Many such difficulties are referred to as “wicked problems,” which won’t be solved with facts alone.

According to John Camillus, writing in Harvard Business Review, environmental degradation, terrorism and poverty are all classic examples of wicked problems. Wicked problems have innumerable causes, are interconnected with other problems and rarely have single acceptable solutions. Hundreds of studies can be carried out, and still the answer can be up in the air as to what should be done.

To solve wicked problems we need to approach science in new, more complex ways. Researchers with different kinds of expertise need to put their heads together. Scientists and decision makers need to interact regularly and become more familiar with each other’s worlds. Citizens and laypeople need to be involved in the research.

This new kind of science goes under various names: citizen science, community-based participatory research, science democratization and participatory action research. But, in each case, science is being transformed in ways all of us need to know about because we have important roles to play in making this new approach succeed.

At the heart of these new approaches is the need to move away from what David Cash, a world leader in science-policy analysis, points to as the all-too-common “loading dock” approach to science. This approach has been likened to scientists following the model of a factory where widgets are produced and then trundled out to the loading dock where someone eagerly waits to pick up the supposedly useful product. But the audience for the science product may not be there. We may be creating a product that people struggling with wicked problems like poverty or hunger, for example, can’t use because it is built on science that does not take into account the full set of complications out there in the real world.

We need stakeholder-engaged, solutions-focused, interdisciplinary work if our scarce science resources are to be mobilized to help solve wicked problems.

The problems are interconnected. We know there is hunger and food insecurity at the same time we struggle to address ever-higher rates of obesity and ever-increasing amounts of food waste ending up in landfills. We know that while we try to address the state’s economic problems by encouraging young entrepreneurs to take up our traditional resource industries, the very resources their future will depend on — such as seafood and shellfish beds — are in decline or threatened by polluted runoff.

Traditional studies provide incomplete tools to understand wicked problems of these sorts. But Maine researchers are changing the ways they do research in order to make inroads on such issues.

Under the framework of sustainability, they tackle research on safe beaches and shellfish, for example, by bringing together stakeholders such as harvesters and policymakers with biologists, economists, engineers and even researchers who study how groups can more effectively solve problems together.

Or they take up declines in major resource industries such as Maine’s blueberries, which face the prospect of collapsing pollinator bee populations, and they work with stakeholders to create tools such as the BeeMapper software to bring together solutions-focused information often treated independently. Researchers working with the University of Maine’s Senator George J. Mitchell Center for Sustainability Solutions are focusing on this style of complex collaborative research, which is leading to many payoffs.

Not all scientists think the democratization of science is a good thing. Some scientists claim anyone who lacks formal training as a scientist can’t do good research. Some insist that only their discipline does the science right. Some view any science built on citizen science or partnership approaches as second rate. To them it smacks of opinions instead of science.

But we are not talking about going back to the era of matters being decided by opinion instead of scientific results. We are not talking about reverting to times when whoever argued loudest and longest won. Instead, as Roger Pielke teaches us in his highly regarded book, The Honest Broker, a big part of the job of scientists in this new era is to learn how to bring data to decisions and to understand that research is but one piece of an increasingly complex puzzle.

Linda Silka, a social and community psychologist, is a senior fellow at the Senator George J. Mitchell Center for Sustainability Solutions at the University of Maine.

Page Top