Evaluation Design For Complex Global Initiatives Workshop Summary 1st Edition Institute of Medicine All Chapters Instant Download

Full download ebooks at ebookmeta.
com
Evaluation Design for Complex Global Initiatives

Workshop Summary 1st Edition Institute Of Medicine
https://ebookmeta.com/product/evaluation-design-for-complex-
global-initiatives-workshop-summary-1st-edition-institute-
of-medicine/
OR CLICK BUTTON
DOWLOAD NOW
Download more ebook from https://ebookmeta.com

More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Collaboration Between Health Care and Public Health

https://ebookmeta.com/product/collaboration-between-health-care-
and-public-health-workshop-summary-1st-edition-institute-of-
medicine/
Spread Scale and Sustainability in Population Health

https://ebookmeta.com/product/spread-scale-and-sustainability-in-
population-health-workshop-summary-1st-edition-institute-of-
medicine/
Global Health Risk Framework Governance for Global

Health Workshop Summary 1st Edition And Medicine
Engineering National Academies Of Sciences
https://ebookmeta.com/product/global-health-risk-framework-
governance-for-global-health-workshop-summary-1st-edition-and-
medicine-engineering-national-academies-of-sciences/
Financial Incentives to Encourage Development of

Therapies That Address Unmet Medical Needs for Nervous
System Disorders Workshop Summary 1st Edition Institute
Of Medicine
https://ebookmeta.com/product/financial-incentives-to-encourage-
development-of-therapies-that-address-unmet-medical-needs-for-
nervous-system-disorders-workshop-summary-1st-edition-institute-
Global Health Risk Framework Pandemic Financing
Workshop Summary 1st Edition And Medicine Engineering
National Academies Of Sciences
https://ebookmeta.com/product/global-health-risk-framework-
pandemic-financing-workshop-summary-1st-edition-and-medicine-
engineering-national-academies-of-sciences/
Building Health Workforce Capacity Through Community

Based Health Professional Education Workshop Summary
1st Edition Institute Of Medicine Board On Global
Health Global Forum On Innovation In Health
Professional Education Patricia A Cuff
https://ebookmeta.com/product/building-health-workforce-capacity-
through-community-based-health-professional-education-workshop-
summary-1st-edition-institute-of-medicine-board-on-global-health-
global-forum-on-innovation-in-health/
Cross Sector Responses to Obesity Models for Change

Workshop Summary 1st Edition Institute Of Medicine Food
And Nutrition Board Roundtable On Obesity Solutions
Steve Olson
https://ebookmeta.com/product/cross-sector-responses-to-obesity-
models-for-change-workshop-summary-1st-edition-institute-of-
medicine-food-and-nutrition-board-roundtable-on-obesity-
solutions-steve-olson/
Exploring Shared Value in Global Health and Safety

Workshop Summary 1st Edition And Medicine Engineering
National Academies Of Sciences
https://ebookmeta.com/product/exploring-shared-value-in-global-
health-and-safety-workshop-summary-1st-edition-and-medicine-
engineering-national-academies-of-sciences/
Preventing Intimate Partner Violence in Uganda Kenya

and Tanzania Summary of a Joint Workshop by the
Institute of Medicine the National Research Council and
the Uganda National Academy of Sciences 1st Edition
National Research Council Institute Of Medicine Board
https://ebookmeta.com/product/preventing-intimate-partner-
violence-in-uganda-kenya-and-tanzania-summary-of-a-joint-
On Global Health Forum On Global Violence Prevention
workshop-by-the-institute-of-medicine-the-national-research-
Louise Flavahan
THE NATIONAL ACADEMIES PRESS 500 Fifth Street,
NW Washington, DC 20001
NOTICE: The workshop that is the subject of this workshop

summary was approved by the Governing Board of the National
Research Council, whose members are drawn from the councils of
the National Academy of Sciences, the National Academy of
Engineering, and the Institute of Medicine.
This activity was supported by contracts between the National

Academy of Sciences and the Bill & Melinda Gates Foundation, the
Doris Duke Charitable Foundation, the Wellcome Trust, and the
William and Flora Hewlett Foundation. The views presented in this
publication do not necessarily reflect the views of the organizations
or agencies that provided support for the activity.
International Standard Book Number-13: 978-0-309-30258-6

International Standard Book Number-10: 0-309-30258-7
Epub ISBN: 0-309-30261-7
Additional copies of this workshop summary are available for sale

from the National Academies Press, 500 Fifth Street, NW, Keck 360,
Washington, DC 20001; (800) 624-6242 or (202) 334-3313;
http://www.nap.edu.
For more information about the Institute of Medicine, visit the IOM
home page at: www.iom.edu.
Copyright 2014 by the National Academy of Sciences. All rights

reserved.
Printed in the United States of America
The serpent has been a symbol of long life, healing, and knowledge
among almost all cultures and religions since the beginning of
recorded history. The serpent adopted as a logotype by the Institute
of Medicine is a relief carving from ancient Greece, now held by the
Staatliche Museen in Berlin.
Suggested citation: IOM (Institute of Medicine). 2014. Evaluation

design for complex global initiatives: Workshop summary.
Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and
Medicine
The National Academy of Sciences is a private, nonprofit, self-

perpetuating society of distinguished scholars engaged in scientific
and engineering research, dedicated to the furtherance of science
and technology and to their use for the general welfare. Upon the
authority of the charter granted to it by the Congress in 1863, the
Academy has a mandate that requires it to advise the federal
government on scientific and technical matters. Dr. Ralph J. Cicerone
is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964,

under the charter of the National Academy of Sciences, as a parallel
organization of outstanding engineers. It is autonomous in its
administration and in the selection of its members, sharing with the
National Academy of Sciences the responsibility for advising the
federal government. The National Academy of Engineering also
sponsors engineering programs aimed at meeting national needs,
encourages education and research, and recognizes the superior
achievements of engineers. Dr. C. D. Mote, Jr., is president of the
National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National

Academy of Sciences to secure the services of eminent members of
appropriate professions in the examination of policy matters
pertaining to the health of the public. The Institute acts under the
responsibility given to the National Academy of Sciences by its
congressional charter to be an adviser to the federal government
and, upon its own initiative, to identify issues of medical care,
research, and education. Dr. Harvey V. Fineberg is president of the
Institute of Medicine.
The National Research Council was organized by the National
Academy of Sciences in 1916 to associate the broad community of
science and technology with the Academy’s purposes of furthering
knowledge and advising the federal government. Functioning in
accordance with general policies determined by the Academy, the
Council has become the principal operating agency of both the
National Academy of Sciences and the National Academy of
Engineering in providing services to the government, the public, and
the scientific and engineering communities. The Council is
administered jointly by both Academies and the Institute of
Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and
vice chair, respectively, of the National Research Council.
www.national-academies.org
PLANNING COMMITTEE FOR THE WORKSHOP
ON
EVALUATING LARGE-SCALE, COMPLEX,
MULTI-NATIONAL GLOBAL HEALTH
INITIATIVES1
ANN KURTH (Chair), New York University, New York, NY

GEORGE ALLEYNE, Pan American Health Organization,
Washington, DC
KARA HANSON, London School of Hygiene & Tropical Medicine,
London, United Kingdom
DOROTHY MUROKI, FHI 360, Nairobi, Kenya
JONATHON SIMON, Boston University, Boston, MA
MARTIN VAESSEN, ICF International, Rockville, MD
IOM Staff
BRIDGET B. KELLY, Project Co-Director
KIMBERLY A. SCOTT, Project Co-Director
KATE MECK, Associate Program Officer
CHARLEE ALEXANDER, Senior Program Assistant (from November
2013)
JULIE WILTSHIRE, Financial Officer
PATRICK W. KELLEY, Senior Board Director, Board on Global
Health
_____________
1 Institute of Medicine planning committees are solely responsible for organizing
the workshop, identifying topics, and choosing speakers. The responsibility for the
published workshop summary rests with the workshop rapporteur and the
institution.
Reviewers
This workshop summary has been reviewed in draft form by

individuals chosen for their diverse perspectives and technical
expertise, in accordance with procedures approved by the National
Research Council’s Report Review Committee. The purpose of this
independent review is to provide candid and critical comments that
will assist the institution in making its published workshop summary
as sound as possible and to ensure that the workshop summary
meets institutional standards for objectivity, evidence, and
responsiveness to the study charge. The review comments and draft
manuscript remain confidential to protect the integrity of the
process. We wish to thank the following individuals for their review
of this workshop summary:
RICHARD BERK, University of Pennsylvania

ANN KURTH, New York University
DOROTHY MUROKI, FHI 360,
SANJEEV SRIDHARAN, University of Toronto
HOWARD WHITE, International Initiative for Impact Evaluation
(3ie)
Although the reviewers listed above have provided many

constructive comments and suggestions, they did not see the final
draft of the workshop summary before its release. The review of this
workshop summary was overseen by Enriqueta Bond, President
Emeritus, Burroughs Wellcome Fund. Appointed by the Institute of
Medicine, she was responsible for making certain that an
independent examination of this workshop summary was carried out
in accordance with institutional procedures and that all review
comments were carefully considered. Responsibility for the final
content of this workshop summary rests entirely with the rapporteur
and the institution.
Acknowledgments
The planning committee and project staff deeply appreciate many

valuable contributions from individuals who assisted us with this
project. First, we offer our profound thanks to all of the presenters
and discussants at the workshop, who gave so generously of their
time and expertise. These individuals are listed in full in the
workshop agenda in Appendix B. We are also grateful to the many
participants who attended the workshop both in person and via the
live webcast. The engagement of all those in attendance was robust
and vital to the success of the event. We are also particularly
appreciative of the thoughtful and creative contributions of Mary
Ellen Kelly and James Kelly, who applied their many years of
experience to help generate the fictional initiative used for the
hypothetical design exercise at the workshop.
In addition, we thank the sponsors of this project for their
support: the Bill & Melinda Gates Foundation, the Doris Duke
Charitable Foundation, the Wellcome Trust, and the William and
Flora Hewlett Foundation. We are also grateful to the Wellcome Trust
for hosting the workshop, with a special thanks to Zoe Storey,
Danielle Taplin, and all of the staff there for their gracious assistance
in support of every aspect of the event. We also extend many thanks
to Anthony Mavrogiannis and the staff at Kentlands Travel for
supporting the travel needs and requirements of this project. We
appreciate LeAnn Locher’s creative work in designing the report
cover. Finally, we convey our gratitude for the hard work of the many
staff of the Institute of Medicine and the National Academies who
supported the project at every stage, from its inception to the
workshop to the final production of this workshop summary report.
Contents
1 INTRODUCTION AND OVERVIEW OF THE WORKSHOP

Terms and Objectives
Messages from the Workshop
Organization of the Workshop Report
2 OVERVIEW FRAMEWORK FOR COMPLEX EVALUATIONS

The BetterEvaluation Initiative Rainbow Framework
Navigating the Framework
3 FRAMING THE EVALUATION

Reflections from the Experience of the Evaluation of the U.S.
President’s Malaria Initiative
Evaluation at the United Nations
An Evaluation Funder’s Perspective on Where Problems Arise
Lessons Learned from the Global Fund to Fight AIDS,
Tuberculosis, and Malaria
Comparing Evaluations of Selected Global Initiatives
Impact of an Evaluation
Other Topics Raised in Discussion
4 DEVELOPING THE EVALUATION DESIGN AND SELECTING

METHODS
Institute of Medicine Evaluation of PEPFAR
Global Fund Evaluation
Affordable Medicines Facility–Malaria Assessment
The Search for Good Practice in Complex Evaluations
5 MAPPING DATA SOURCES AND GATHERING AND ASSESSING
DATA
Data Mapping in the IOM Evaluation of PEPFAR
Data Issues in the Global Fund Evaluation
Data Approach and Challenges in the Affordable Medicines
Facility–Malaria Evaluation
President’s Malaria Initiative Evaluation
Using Routine Program Data
Working with Financial Data
Developing a Large-Scale Data Infrastructure
6 APPLYING QUALITATIVE METHODS TO EVALUATION ON A

LARGE SCALE
Rigor and Credibility in Qualitative Design
The Value of Qualitative Methods
7 APPLYING QUANTITATIVE METHODS TO EVALUATION ON A

LARGE SCALE
Outcomes Matter
Mathematical Modeling as a Tool for Program Evaluation
Extended Cost-Effectiveness Analysis
8 ANALYSIS THROUGH TRIANGULATION AND SYNTHESIS TO

INTERPRET DATA IN A MIXED METHODS EVALUATION
Triangulation in the Global Environmental Facility
Developing a Deeper and Wider Understanding of Results
Triangulation in Practice
Approaches to Triangulation
Lessons from the IOM’s PEPFAR Evaluation
9 EVOLVING METHODS IN EVALUATION SCIENCE

Use of Realist Methods to Evaluate Complex Interventions and
Systems
Innovative Designs for Complex Questions
Comparative Systems Analysis: Methodological Challenges and
Lessons from Education Research
10 LESSONS FROM LARGE-SCALE PROGRAM EVALUATION ON A

NOT-QUITE-AS-LARGE SCALE
Saving Mothers, Giving Life Strategic Implementation Evaluation
Avahan—Reducing the Spread of HIV in India
EQUIP—Expanded Quality Management Using Information
Power
11 USING EVALUATION FINDINGS AND COMMUNICATING KEY

MESSAGES
The U.S. President’s Malaria Initiative
Roads to a Healthy Future
The South African Perspective on Influencing Policy and
Performance
Communicating Results from the PEPFAR Evaluation
12 ENVISIONING A FUTURE FOR EVALUATION

Framing the Evaluation
Building an Ecology of Evidence
Capacity Building for Future Evaluators
REFERENCES
APPENDIXES
A STATEMENT OF TASK
B WORKSHOP AGENDA
C PARTICIPANT BIOGRAPHIES
D EVALUATION INFORMATION SUMMARY FOR CORE EXAMPLE
INITIATIVES
E EVALUATION DESIGN RESOURCES HIGHLIGHTED AT THE
WORKSHOP
1
Introduction and Overview of the Workshop1
Every year, public and private funders spend many billions of

dollars on large-scale, complex, multi-national health initiatives. The
only way to know whether these initiatives are achieving their
objectives is through evaluations that examine the links between
program activities and desired outcomes. Investments in such
evaluations, which, like the initiatives being evaluated, are carried
out in some of the world’s most challenging settings, are a relatively
new phenomenon. As such, it is worthwhile to reflect on the
evaluations themselves to examine whether they are reaching
credible, useful conclusions and how their performance can be
improved.
In the last 5 years, evaluations have been conducted to determine
the effects of some of the world’s largest and most complex multi-
national health initiatives. On January 7–8, 2014, the Institute of
Medicine (IOM) held a workshop at the Wellcome Trust in London to
explore these recent evaluation experiences and to consider the
lessons learned from how these evaluations were designed, carried
out, and used. The statement of task for the workshop can be found
in Appendix A. The workshop brought together more than 100
evaluators, researchers in the field of evaluation science, staff
involved in implementing large-scale health programs, local
stakeholders in the countries where the initiatives are carried out,
policy makers involved in the initiatives, representatives of donor
organizations, and others to derive lessons learned from past large-
scale evaluations and to discuss how to apply these lessons to future
evaluations. The workshop was sponsored by the Bill & Melinda
Gates Foundation, the Doris Duke Charitable Foundation, the
Wellcome Trust, and the William and Flora Hewlett Foundation.
This workshop did not attempt to provide a comprehensive review
of the rich body of literature on program evaluation theory or
practice (Berk and Rossi, 1999; Leeuw and Vaessen, 2009; Rogers,
2008; Rossi et al., 2004; Royse et al., 2009; Stern et al., 2012; White
and Phillips, 2012), but the evaluation examples that were examined
drew on an expansive array of the available evaluation
methodologies and applied them in different ways to the large-scale,
complex, multi-national health initiatives. As a result, they have
produced a large body of experience and knowledge that can benefit
evaluations of health and development initiatives. The workshop
looked at transferable insights gained across the spectrum of
choosing the evaluator, framing the evaluation, designing the
evaluation, gathering and analyzing data, synthesizing findings and
recommendations, and communicating key messages. The workshop
explored the relative benefits and limitations of different quantitative
and qualitative approaches within the mixed methods designs used
for these complex and costly evaluations. It was an unprecedented
opportunity to capture, examine, and disseminate expert knowledge
in applying evaluation science to large-scale, complex programs.
This workshop report summarizes the presentations and
discussions at the workshop and is intended to convey what
transpired to those involved or affected by large-scale, multi-national
health initiatives, including implementers, stakeholders, evaluators,
and funders of initiatives and evaluations.
TERMS AND OBJECTIVES
In her opening remarks at the workshop, Ann Kurth, professor of

nursing, medicine, and public health at New York University and
chair of the planning committee for the workshop, offered how the
terms used in the workshop’s name were being applied:
Large-scale—The total cumulative budgets over multiple

years amounting to at least hundreds of millions of U.S.
dollars
Multi-national—Implementation on a global scale, including
multiple countries and regions or subregions of the world
Complex
Encompassing multiple components, such as varied types of

interventions and programs implemented in varied settings,
systems-strengthening efforts, capacity building, and efforts
to influence policy change
Implementation at varied levels within partner countries
through a large number of diverse, multisectoral partners,
including an emphasis on local governments and
nongovernmental institutions
Evaluations of four large-scale, complex, multi-national health

initiatives acted as core examples for the workshop:
Global Fund to Fight AIDS, Tuberculosis, and Malaria (GF)

(Sherry et al., 2009)
U.S. President’s Malaria Initiative (PMI) (Simon et al., 2011)
Affordable Medicines Facility–malaria (AMFm) (Tougher et al.,
2012)
U.S President’s Emergency Plan for AIDS Relief (PEPFAR)
(IOM, 2013)
Appendix C provides a comparison of the evaluations for these

initiatives. In addition, the workshop examined other evaluations of
large-scale health and development initiatives along with smaller-
scale evaluations that share similar features of complexity.
Evaluations need to be credible, rigorous, feasible, affordable, and
matched to the priority evaluation questions, aims, and audiences,
Kurth said. No single evaluation design can serve every purpose, and
every evaluation must make strategic choices that fit its context and
goals. But the process of designing and conducting an evaluation
has key strategic decision-making points, and the available choices
have different advantages and disadvantages that result in trade-offs
for any given design decision. Evaluations of complex initiatives
require more complicated strategic design considerations, but many
of the issues discussed at the workshop are applicable to evaluations
all along the spectrum of complexity.
Though the workshop sought to identify lessons learned, it was
not designed to look backward, said Kurth. Rather, the underlying
objective was to be “candid, open, and frank” about past
experiences to create a foundation for future improvements.
MESSAGES FROM THE WORKSHOP
In the final session of the workshop, some of the important

messages over the previous 2 days were recapitulated by three
experienced evaluators (Chapter 12 provides a full account of their
remarks):
1. Sanjeev Sridharan, director of the Evaluation Centre for

Complex Health Interventions at Li Ka Shing Knowledge
Institute at St. Michaels Hospital and associate professor in
the Department of Health Policy, Management, and
Evaluation at the University of Toronto;
2. Charlotte Watts, head of the Social and Mathematical
Epidemiology Group and founding director of the Gender,
Violence, and Health Centre in the Department for Global
Health and Development at the London School of Hygiene
and Tropical Medicine; and
3. Elliot Stern, emeritus professor of evaluation research at
Lancaster University and visiting professor at Bristol
University.
The workshop then closed with reflections from representatives of

the four funders of the workshop— Gina Dallabetta of the Bill &
Melinda Gates Foundation, Mary Bassett of the Doris Duke Charitable
Foundation, Jimmy Whitworth of the Wellcome Trust, and Ruth
Levine of the William and Flora Hewlett Foundation—on the major
lessons and messages they were taking away from the event.
The following messages of the workshop are drawn from these
speaker’s remarks. These should not be seen as recommendations
or conclusions emerging from the workshop, but they provide a
useful summary of some of the major topics discussed.
What Evaluations Can Do

Evaluations typically have multiple objectives, said Charlotte
Watts, head of the Social and Mathematical Epidemiology Group and
founding director of the Gender, Violence, and Health Centre in the
Department for Global Health and Development at the London
School of Hygiene and Tropical Medicine. Some evaluations are
focused specifically on assessing an intervention’s impact and cost-
effectiveness, but others have broader public good aspects. An
evaluation may also aim to derive lessons about scaling up or
replicating effective interventions, build capacity for evaluations, or
strengthen networks of researchers and practitioners.
Ruth Levine, director of the Global Development and Population
Program at the William and Flora Hewlett Foundation Evaluations,
commented that evaluations are used to hold governments, funders,
and other stakeholders accountable for the use of the resources that
are dedicated to large initiatives that have proliferated and have high
political visibility. Similarly, Jimmy Whitworth, head of population
health at the Wellcome Trust, notes that policy makers have been
challenging the public health community to learn more about the
effects of its interventions as a way to justify and increase
investments in large-scale initiatives. To that end, evaluations of
public health investments may inform not only program
improvements, but also policy and funding decisions.
Sanjeev Sridharan, director of the Evaluation Centre for Complex
Health Interventions at Li Ka Shing Knowledge Institute at St.
Michaels Hospital and associate professor in the Department of
Health Policy, Management, and Evaluation at the University of
Toronto observed that very few large-scale, complex, multi-national
initiatives are well formed from their earliest stages and noted that
evaluations can also contribute to the development of an initiative.
This may require a changing relationship with evaluators over time,
but it can build capacity in both the evaluation and the initiative that
can lead to continual improvements.
Governance and Evaluator Independence

Gina Dallabetta, a program officer at the Bill & Melinda Gates
Foundation, appreciated that the workshop focused on the larger
view of evaluation, including issues such as governance. In
particular, the question of how independent evaluators should be
was raised by several of the participants. These initiatives are
incentivized to claim success so that they can maintain high levels of
resources and political commitment, Levine noted. On the other
hand, Sridharan noted that program staff are generally among the
most critical observers of their programs. It does not take a faraway
researcher to be objective. Sridharan proposed a nuanced position
with degrees of independence depending on the phase and intent of
the evaluation. For an evaluation early in a project, a close
relationship with an evaluator may allow for valuable input to
program staff as they design or modify an intervention. A results-
focused evaluation may need to achieve more independence from a
program to deliver unbiased results. Elliot Stern, emeritus professor
of evaluation research at Lancaster University and visiting professor
at Bristol University, added that it may be possible to have different
people involved in different evaluation phases to obtain the
appropriate levels of independence.
Evaluation Framing and Design
Evaluations need to prioritize the questions they are asking, said
Levine, which means thinking through the kinds of questions that
could change the minds of decision makers, whether within the
program or at a higher level. Watts stated that understanding
program effectiveness cannot be reduced to answering a closed-
ended question about whether “it worked.” Perhaps the evaluation
questions should be more nuanced: Can you do it at scale? Can you
do it with this population? Can you sustain it? This provides a
greater space for the framing of questions, the evaluation design,
and for partnership between evaluators and program staff.
A wealth of techniques and methodologies are available for
evaluation, but the strength of an evaluation lies in careful design.
Many of the speakers highlighted that an underlying initial step is to
articulate an underlying program theory or similar framework to
understand the fundamental assumptions that need to be
interrogated. Watts emphasized the importance of designing a mixed
methods approach to achieve a full understanding of a large-scale,
complex, multi-national initiative. Qualitative and quantitative work
can be nested in parallel, and qualitative and quantitative data can
be triangulated to increase confidence in the evidence base for
evaluation findings.
Understanding Context
The evaluators emphasized the critical importance for evaluation
design of understanding the relationship between context and the
desired outcomes for intervention and evaluation designs. Sridharan
noted it is best to bring the knowledge of context in at the start, but
also to understand how it is evolving and adapting over time. Mary
Bassett of the Doris Duke Charitable Foundation Evaluators
reiterated the need to devote heightened attention to context,
referring to how Elliot Stern in his comments “really challenged us to
unpack the notion of what context means.” Contextual issues arise
on micro-, meso-, and macro-scales, and all can be important.
Droughts, economic crises, and political changes are some factors
that can affect the outcome of an initiative and should be tracked,
but it is also important to think about how to understand issues of
leadership, power, trust, communication, and community
engagement that have all been talked about, Bassett said.
Data Availability and Quality

Many types of data collection can be prospectively embedded
within a program for evaluation. Gina Dallabetta, a program officer
at the Bill & Melinda Gates Foundation, emphasized strengthening
capacity for quality data collection in countries, especially as projects
become larger and more complex. Dallabetta noted that program
evaluations can be hampered by a lack of quality routine data
collected within countries, reflecting a need for management
expertise to help countries collect better data, including process
data, outcome data, and financial data. These data can be used for
both program monitoring by management and evaluation, but data
collection and analysis need to be based on a theory of change
before program implementation begins, said Dallabetta.
However, evaluators also need to do original data collection to
have ways to validate the data reported by programs, Levine said.
She also observed that many more sources of nontraditional data
will be available in the future, such as geospatial analyses, citizen
feedback, transactional data about what people are purchasing and
where they are going, and sensors such as smart pill bottles.
All four of the workshop funders emphasized the importance of
open data, so that the information on which conclusions are based is
available to others. Levine noted that if data can be made available
to others for reanalysis, this can reinforce technical quality.
Dallabetta added, however, that data sometimes belong to
governments or have multiple owners, which may require that one
centralized place exist in a country where people can view data.
Maintaining open data also requires work, such as data archiving
and documentation, that donors need to build into their funding.
Using the Results of Evaluations
Though the use of an evaluation’s results can be one of the factors
least in control of an evaluation team, evaluators can enhance the
use of their results in a variety of ways, said Levine. An especially
promising approach is to meaningfully engage a wide range of
stakeholders in an ongoing way. Evaluators also can encourage
systematic follow-up of recommendations, in part by creating a
culture of learning versus one of punishment. Evaluations need
adequate planning, skills, and budgets for a fit-for-purpose
dissemination, Levine said. Whitworth observed that the public
health community also needs to do a better job of celebrating and
publicizing its successes as a way of increasing support for large-
scale programs. Large-scale, complex, multi-national initiatives have
produced some of the biggest success stories of international health
and development assistance, and those stories have been backed up
by credible evidence, said Levine. Watts similarly observed that
strong evaluations require resources, commitment, investments,
trust, and strong relationships, but they can be tremendously
beneficial for public health.
Final Reflections on Future Large-Scale, Complex Evaluations

As part of the workshop’s final session, Levine shared some
thoughts about future evaluations of large-scale, complex, multi-
national initiatives as well as other evaluations that will benefit from
the information shared at the workshop.
One important lesson derived from recent large-scale initiatives is
how to increase the space for serious evaluations, said Levine. The
public health community has a tradition of basing program design on
good evidence and then learning as it goes based on additional
evidence. “The potential for evaluations to actually make a
difference is there,” said Levine, also observing that improving the
technical quality of evaluations is a demanding task.
One important trend that will influence future evaluations is a new
partnership model with the countries in which programs are being
implemented and evaluated. Evaluations need mandates from
governments and donors doing rigorous work, Levine observed, and
they need the funding to be able to do that work. At the same time,
evaluations need the governance and advisory structures to be
insulated from political influence.
The advocacy community can support evaluations in this regard by
praising initiatives that not only do evaluations but then make use of
findings to correct shortcomings. “The very same advocates who are
so good at pushing for more money for global health programs can,
and sometimes have been, very capable advocates for evaluating
and using the findings from evaluations for more effective
programming,” said Levine.
Another trend that will make itself felt in the future is an increase
in “uninvited co-evaluators.” Many people have access to evaluation
information who have an interest in challenging not just the program
but the evaluation. As Levine said, “There is a lower barrier to entry
into the conversation, and that is in some abstract way a positive
thing, [but] in the day-to-day reality, it’s very challenging.”
Finally, Levine closed with a list of potential activities or steps for
improvement for evaluators and funders:
Evaluators
Document the stories of evaluations.

Create greater value in global collaborations.
Be honest about the feasibility of sponsors’ demands.
Participate in method development and validation.
Connect with the evaluation community outside of the health
sector.
Train the next generation.
Embrace transparency.
Funders
Create incentives for learning.
Make reasonable demands of evaluators, and fund at the
right level.
Permit or require transparency.
ORGANIZATION OF THE WORKSHOP REPORT
Following this introductory chapter, Chapter 2 of this summary of

the workshop describes an overview framework for evaluation
design, introducing many of the topics discussed at the workshop.
Chapters 3–8 are arranged to present the major components of
design and implementation in roughly the sequence that they might
be addressed during the course of an evaluation, although each
evaluation is different and many of these components are typically
addressed or re-addressed iteratively and continuously throughout
the evaluation process.
Chapter 3 discusses how evaluations are framed when choosing
the evaluator, establishing the governance structure of the
evaluation, and developing and prioritizing the evaluation questions.
Chapter 4 examines the development of an evaluation’s design and
particularly the methodological choices and trade-offs evaluators
must make in the design process. Chapter 5 considers data sources
and the processes of gathering and assessing data.
Chapters 6 and 7, which are drawn from two of four concurrent
sessions that were held during the workshop, examine
methodological and data issues in more detail. Chapter 6 looks at
the application of qualitative methods to evaluation on a large scale,
while Chapter 7 does the same for quantitative methods. Chapter 8
then turns to the use of triangulation and synthesis in analyzing data
from multiple data sources and across multiple methods to yield a
deeper and richer perspective on an initiative and increase
confidence in the evidence base for evaluation findings.
Chapters 9 and 10, which are drawn from the other two workshop
concurrent sessions, explore specific extensions of some of the ideas
discussed earlier in the workshop. Chapter 9 looks at evolving
methods in evaluation science, including realist methods and
nonexperimental, observational, and mixed methods. Chapter 10
discusses how principles that are important for large-scale program
evaluations can similarly be applied to complex evaluations on a
smaller scale. Chapter 11 then examines how the findings and key
messages of an evaluation are used and can be disseminated to
diverse audiences. In Chapter 12, three experienced evaluators
reflect on the messages of the workshop and how they might apply
to future evaluations through the lens of a hypothetical evaluation
design exercise for a fictional multi-national initiative.
_____________
1 The planning committee’s role was limited to planning the workshop. The
workshop summary has been prepared by the workshop rapporteur (with the
assistance of Charlee Alexander, Bridget Kelly, Kate Meck, and Kimberly Scott) as a
factual summary of what occurred at the workshop. Statements,
recommendations, and opinions expressed are those of individual presenters and
participants; they are not necessarily endorsed or verified by the Institute of
Medicine, and they should not be construed as reflecting any group consensus.
2
Overview Framework for Complex Evaluations
Important Points Made by the Speaker
Interventions can vary in complexity along different

dimensions.
Many methods are available to describe the
implementation of an intervention, to acquire data about
an intervention’s effects, and to assess the contribution
to or attribution of impact.
Many methods are complex and rapidly changing,
creating a demand for guidance in their use.
In the opening session of the workshop, Simon Hearn, a research

fellow at the Overseas Development Institute, set the stage for
discussing the practice of designing and implementing mixed
methods evaluations of large complex interventions and presented
an overall framework for approaching these types of evaluations. He
described the BetterEvaluation Initiative, a global collaboration of
many organizations dedicated to improving evaluation practice and
theory.1
Hearn identified three major challenges in evaluating complex
interventions:
1. Describing what is being implemented
2. Getting data about impacts
3. Attributing impacts to a particular program
Many different evaluation methods are available to take on these

challenges, as are guides for how to do evaluations. But many
methods are evolving rapidly, and new methods and data sources
are becoming available, such as the use of mobile phones or social
media. Also, options exist for understanding causality that do not
involve experimental methods, though these are often not covered in
guides. In such circumstances, evaluators often use the methods
they have always used, said Hearn, which may or may not be suited
to the problem at hand.
Hearn also discussed six facets of interventions that, when viewed
together, can be used to gauge the complexity of an intervention:
1. Objectives: Are they clear and defined ahead of time, or are

they emergent and changing over time?
2. Governance: Is it clear and defined ahead of time, or is it
characterized by shifting responsibilities and networks of
actors?
3. Implementation: Is the implementation of the intervention
consistent across places and across time, or will it shift and
change over time?
4. Necessariness: Is the intervention the only thing necessary
for its intended impacts, or do other pathways lead to the
impacts, and are these pathways knowable?
5. Sufficiency: Is the intervention sufficient to achieve the
impacts, or do other factors need to be in place for the
impacts to be realized, and are these other factors
predictable before the fact?
6. Change trajectory: Is the change trajectory
straightforward? Can the relationship between inputs and
outcomes be defined, and will this relationship change over
time?
THE BETTEREVALUATION INITIATIVE RAINBOW
FRAMEWORK
Hearn and his colleagues have developed the BetterEvaluation

Rainbow Framework to help evaluators navigate the choices
available at each stage of an evaluation (see Figure 2-1). The
framework organizes clusters of tasks associated with each stage of
the evaluation process, although the stages and tasks are not
necessarily sequential, and each is as valuable as the other.
Furthermore, all are subject to management, which acts as a sort of
prism through which all the different stages of an overall evaluation
process are viewed.
The first task is to define what is to be evaluated. This involves
developing an initial description of the initiative or program being
evaluated (the evaluand), developing a program theory or logic
model to describe how the program is intended to create change,
and identifying potential unintended results. For example, a logic
model may consist of a simple pipeline or results chain, a more
sophisticated logical framework, more structured and free-flowing
outcomes hierarchies, or realist matrices, though complex
evaluations are much more likely to rely on the more sophisticated
models. Prominent questions include whether an evaluation is
looking at the effects on policy, the effects on populations, or both;
whether multiple levels of activity are being evaluated; and who the
stakeholders are. A clear description can engage and inform all of
the stakeholders involved in the evaluation, Hearn said, which is
particularly important in complex evaluations where many people are
involved.
FIGURE 2-1 BetterEvaluation Initiative’s Rainbow Framework, as presented by
Hearn, separates an evaluation into equally important tasks that each need to be
managed.
SOURCE: BetterEvaluation, 2014.
The second task is to frame what is to be evaluated. Framing an

evaluation is necessary to design that evaluation. Framing involves
identifying the primary intended users, deciding on the purposes of
the evaluation, specifying key evaluation questions, and determining
what “success” would look like—what standards or criteria will be
used to make judgments about the program? Complex interventions
are likely to have multiple contributors and users of results, and
Hearn noted that the different purposes of these users can conflict.
Users also may have different evaluation questions that need to be
addressed, which means they might have different understandings
of success.
The third task is to describe what happened. This task involves the
use of samples, measures, or indicators; the collection and
management of data; the combination of qualitative and quantitative
data; the analysis of data; and the visualization and communication
of data. For example, in Chapter 6.) Are data from one phase of a
project being used to inform another phase? Are the data being
gathered in parallel or in sequence? Many different options are
available in this area, said Hearn.
The fourth task is to understand the causes of outcomes and
impacts. What caused particular impacts, and did an intervention
contribute to those outcomes? Do the results support causal
attributions? How do the results compare with the counterfactual
analysis? What alternative explanations are possible? Simply
collecting information about what happened cannot answer
questions about causes and effects, whereas an evaluation must
deal with causation in some way, Hearn observed. Evaluations often
either oversimplify or overcomplicate this process. Oversimplification
can come from an implicit assumption that if something is observed
it can be understood as caused by the program or interventions—
making a “leap of faith” without doing the analysis to verify the
claim. Overcomplication can lead to overly elaborate experimental
designs or to the analysis of very large datasets with overly
sophisticated statistical techniques. Experimental designs can be
powerful, but other options are also available, and causation can be
measured even without control groups or experimentation. Theory-
based designs, participatory designs, counterfactual analysis,
regulatory frameworks, configurational frameworks, generative
frameworks, realist evaluation, general elimination method, process
tracing, contribution analysis, and qualitative comparative analysis
are among the many techniques that can explore causation. Indeed,
said Hearn, the BetterEvaluation website has 29 different options for
understanding causes.
The fifth task is to synthesize data to make overall judgments
about the worth of an intervention. Among the many questions that
can be asked at this stage are: Was it good? Did it work? Was it
effective? For whom did it work? In what ways did it work? Did it
provide value for money? Was it cost-effective? Synthesis can occur
at the micro level, the meso level, and the macro level. At the micro
level, performance on particular dimensions is assessed. At the meso
level, different individual assessments can be synthesized to answer
evaluation questions. At the macro level, the merit or worth of an
intervention can be assessed.
Synthesis can look at a single evaluation or at multiple
evaluations, and it can generalize findings from, for example, a small
population to a larger population. Synthesis can be difficult in cases
where some positive and some negative impacts have been
achieved, which requires weighing up the strengths and weaknesses
of the interventions. But synthesis is essential for evaluations, said
Hearn, even though it is often slighted or overlooked in textbooks
and research designs.
The sixth task is to report results and support use. “We are in the
business of evaluation because we want those evaluations to make a
difference,” said Hearn. “We do not want them just to be published
as a report and for the users of those reports to ignore them or to
misuse them.” This task requires identifying reporting requirements
for different stakeholders; developing reporting media, whether
written reports, social media campaigns, or some other output;
ensuring accessibility for those who can use the results; developing
recommendations where appropriate; and helping users of
evaluations to apply the findings.
Finally, Hearn discussed the management of these six tasks, which
includes but is not limited to the following elements:
Understand and engage with stakeholders.

Establish decision-making processes.
Decide who will conduct the evaluation.
Determine and secure resources.
Define ethical and quality evaluation standards.
Document management processes and agreements.
Develop an evaluation plan or framework.
Review the evaluation.
Develop evaluation capacity.
These management tasks are relevant throughout the entire process

of the evaluation, applying to each of the previous tasks.
NAVIGATING THE FRAMEWORK
Hearn provided three tips to help evaluators navigate the

framework, which is available on the BetterEvaluation website.2 The
first tip is to look at the types of questions being asked, whether
descriptive, causal, synthetic, or use oriented. For example, a
descriptive question is whether the policy was implemented as
planned; a causal question is whether a policy contributed to
improved health outcomes; a synthetic question is whether the
overall policy was a success; and a use-oriented question is what
should be done. The question being asked will determine which part
of the framework to access.
The second tip is to compare the pros and cons of each possible
method. The website provides methods and resources for each of
the six tasks, and this information can be used to select the optimal
method.
The third tip is to create a two-dimensional evaluation matrix that
has the key evaluation questions along one side and the methods
along the other. By filling out this matrix, a toolkit can be developed
to match questions with the methods that will be used to answer
those questions.
The website offers much more information on each of the
elements of the framework, as well as other resources. The
BetterEvaluation organization also runs events, clinics, workshops,
and other events to help teams work through evaluation design, and
it then feeds the information generated by such experiences into its
website. In addition, it issues publications and other forms of
guidance and information.
The vision of the BetterEvaluation initiative, Hearn concluded, is to
foster collaborations to produce more information and more
guidance on methods to improve evaluation. The topic and the
structure of this workshop are aligned to the principles of the
framework, Hearn observed, and it is an opportunity to “push us
further” to fill gaps and work together for a “better understanding of
the richness and variety of methods” for evaluation.
_____________
1
More information is available at http://betterevaluation.org
(accessed April 7, 2014).
2
More information is available at http://betterevaluation.org
(accessed April 7, 2014).
3
Framing the Evaluation
Important Points Made by the Speakers
By prioritizing and organizing the questions to be

addressed by evaluations into manageable units, realistic
instruments and a framework for conducting the
evaluation can be designed.
Evaluations require that trade-offs be made along a
number of dimensions, including the balance of
independence and interdependence.
Multiple goals for an evaluation may not be incompatible
but often require different approaches.
Evaluations can enhance their value by building in-
country capacity and by involving more local participants
in the evaluation.
Any evaluation effort starts by framing the evaluation. In the

workshop’s opening session, five panelists discussed various
approaches for this key initial step. From their individual
experiences, the panelists addressed such issues as developing and
prioritizing the evaluation questions, defining the audiences and
intended uses for the evaluation, the relationship between the
evaluators and the evaluands, and trade-offs in choosing the type of
evaluator and in identifying and establishing the governance
structure for an evaluation. Evaluations of large-scale, complex,
multi-national initiatives are themselves going to be complex, which
requires, as pointed out in the previous chapter, well-managed
pursuit of discrete tasks.
REFLECTIONS FROM THE EXPERIENCE OF THE EVALUATION

OF THE U.S. PRESIDENT’S MALARIA INITIATIVE
Framing an evaluation starts with a small set of well-defined

questions, said workshop planning committee member and panel
moderator Jonathon Simon, who is Robert A. Knox Professor and
director of the Center for Global Health and Development at Boston
University. What often happens, however, is that evaluators are
presented with “laundry lists from smart people [who are]
passionately committed to issues within agencies or organizations
that want to know everything about everything.” Simon gave several
examples from the evaluation of the U.S. President’s Malaria
Initiative (PMI), noting that the initial evaluation request from the
PMI included a list of 82 questions that contained another 50 or so
questions nested within that list. It is essential, then, to prioritize
and organize the questions in manageable units that can be used to
design realistic instruments and a framework for conducting the
evaluation.
It is then necessary to consider the audiences for the results of
the evaluation beyond the discrete audience of those in charge of
the effort being evaluated. For the evaluation of PMI, explained
Simon, the Washington Post was an audience, as was a group of
think tanks that had been criticizing the initiative. A large political
audience for the evaluation was more concerned with whether the
PMI was working and less interested in the technical evaluations of
the interventions. Financial considerations were also a factor, given
that the PMI was up for reauthorization.
Within the U.S. Agency for International Development (USAID)
and the Centers for Disease Control and Prevention (CDC)—the
agencies in charge of the PMI—the leaders of both organizations had
substantive interest in the evaluation, but the evaluation goals of the
two agencies were different. An additional consideration was that
the evaluation was not mandated by Congress but was
commissioned by the initiative’s director. For the evaluators, an
important audience was the group of people leading the country-
level efforts and implementing the program on the ground. “Could
we do an evaluation that actually added value or contributed to
national malaria control programs, and could the country personnel
actually benefit or learn from the evaluation?” Recognizing the ways
in which the results will be reported and used and the legitimate
needs of the different audiences for an evaluation points to the
complexity of designing an evaluation, said Simon.
Regarding the relationship between the evaluators and the
evaluand, Simon said that the reason he was asked to conduct the
evaluation was that he was perceived to be independent of the
“malaria mafia.” While that was the case, Simon said that the
evaluation was funded by grants from USAID, CDC, and other
agencies of the U.S. government. To maintain the objectivity of the
evaluation team, Simon insisted on operational independence. “We
took control of the process once the original framing was done, and
we had a set of agreements that the agencies would not see
anything until they received a draft report,” he said. While it was
important to maintain operational independence from the funders of
the evaluation, Simon and his colleagues often had to rely on the
PMI staff to gain the cooperation of the in-country teams. To
illustrate that objectivity required a balance between independence
and interdependence, Simon explained that while the evaluation
team received 167 comments from the funders in response to the
draft report, the evaluators chose which comments to address and
which to reject.
Maintaining the right balance in terms of independence and
interdependence ties into the issue of trade-offs. Simon identified
seven trade-offs that were made in evaluating the PMI. There were
methodological trade-offs in terms of setting the right mix of
qualitative and quantitative methods and the use of data from
routine monitoring programs. Given that the PMI is active in 15
countries, there were geographic trade-offs; in the end, the
evaluators conducted site visits in 5 countries and relied on
telephone interviews in the other 10 countries, resulting in some
degree of selection bias. There were trade-offs in terms of which
technical interventions were assessed from a functional perspective,
such as indoor residual spraying versus bed nets. Another set of
trade-offs involved the priority given to the various audiences,
including political, financial, programmatic, and country-level
audiences. Time and money were not infinite, which also
necessitated trade-offs. Finally, there is the value trade-off between
the perfect design and results that are useful and informative. “How
you do the value trade-off is one of the key challenges that we deal
with in these large, multicountry, complex evaluations,” he said in
closing.
EVALUATION AT THE UNITED NATIONS
The United Nations (UN) Office of Inspection and Oversight (OID),

which is one of three bodies with oversight functions at the UN, is
responsible for evaluating 32 different UN programs and entities that
engage in a wide range of activities, from peacekeeping operations
to humanitarian and environmental programs, explained Deborah
Rugg, director of the Inspection and Evaluation Division at the UN
Secretariat. Her office has 22 professional evaluators on staff, largely
methodologists, and it contracts with external experts for subject-
matter expertise. OID reports through the UN Secretary General to
the 193 member states. The fact that these evaluations are
mandated gives her office both authority and funding, which makes
what Rugg characterized as a huge difference in terms of
participation by and cooperation from staff with the evaluated
programs.
Each year, her office conducts an average of eight assessments
that examine the extent to which a program has been funded, its
size, how many evaluations of the program have been conducted, if
there is a need for evaluation, and if there are any current topical
issues germane to the need for evaluation. For example, many of
the peacekeeping evaluations are based on current political and
contextual issues that need urgent attention as well as the capacity
within that entity to do an evaluation. She noted that at one time
OID evaluations were largely for accountability purposes and offered
little information in terms of value, which meant that evaluations
were largely noncollaborative activities. Today, Rugg and her
colleagues use a partnership model that solicits input on what needs
to be evaluated to answer important operational and functional
questions. This partnership approach has led to increased use of the
evaluation reports, she said.
The issue of independence is an important one at the UN and for
the UN evaluation group, and Rugg pointed to three levels of
independence. Institutional independence means that her group
operates as an independent group outside of a program without an
institutional direct line of report. Operational independence means
that while the evaluation of UN programs is conducted by a UN
office, her group sits outside of the programs that it evaluates.
Behavioral independence refers to an absence of coercion by the
program being evaluated or of a conflict of interest for those
conducting the evaluation. “I have to prove in all of OID’s
evaluations that we are not unduly influenced by the program, or
more importantly, by any specific country,” explained Rugg. Some
programs are evaluated more frequently than others, but on
average, programs can expect to be evaluated about once every 8
years, which she said is a reasonable time frame if there also are
internal embedded evaluations to answer more timely and program-
relevant questions. “That’s one of the trade-offs with these large-
scale, infrequent evaluations is that they can address high-level
issues with a global context, but they cannot drill down as effectively
as one might expect,” she said. “We would like to see more internal
evaluation capacity building so that that can answer specific
questions in a more timely basis.”
To increase utilization of findings, evaluations now start with a 3-
month inception phase in which her office holds conversations with
potential users, reviews prior evaluations, and attempts to develop a
better understanding of how an evaluation can be useful to program
staff as well as to the UN as a whole. After completing an
evaluation, her group works with the evaluated program and
conducts follow-up sessions to check on implementation of any
recommendations suggested by the evaluation or that are mandated
by the member states. A typical evaluation takes about 1 year, which
includes the 3-month inception period followed by 3 months of data
collection, 3 months of analysis and writing, and a 3-month
clearance process. Rugg characterized this as a short period of time
that balances a trade-off between producing actionable and timely
results of a program against depth of experimental and analytic
design.
In terms of the the Joint United Nations Program on HIV/AIDS
(UNAIDS) program, Rugg said that ongoing internal evaluations are
focusing on performance monitoring, while an external, independent
evaluation is conducted every 5 years and a variety of ad hoc special
evaluation studies focus on specific programmatic issues. In
addition, in-country residents in regional offices around the world
work to support the national governments’ evaluations and capacity
building.
AN EVALUATION FUNDER’S PERSPECTIVE ON WHERE

PROBLEMS ARISE
After agreeing with the points that the previous speakers had
made, Christopher Whitty, chief scientific advisor at the United
Kingdom’s Department for International Development (DFID), noted
that, in his view, program officials who work outside of the health
care arena have not historically had much appreciation for the fact
that “good ideas, passionately delivered by people to the highest
quality, may not work.” As a result, outside of health care, not much
value has been placed on evaluation, though he acknowledged that
this situation is changing for the better. Other positive developments,
he said, include the improvement in the methodologies available for
performing complex evaluations and greater acceptance that mixed
methods approaches, or using both quantitative and qualitative
methods for data collection and analysis, are important for
evaluations.
In his role as a commissioner at DFID, Whitty is on the receiving
end of evaluations and has seen a number of outstanding
evaluations over the past few years, including the independent
evaluation of the Affordable Medicines Facility–malaria (AMFm). Most
evaluations, however, have not been outstanding, and he observed
that some of the reasons are on the side of those who request and
fund evaluations. The biggest problem from the donor perspective,
he said, is that those who commission evaluations have multiple
goals for the evaluation that, while not necessarily incompatible,
actually require distinctly different approaches. One goal is to
provide assurance to those who pay for a given program—the British
public in his case—that their money is not being wasted. A second
goal is to check on the efficacy of a program and make course
corrections if needed. The third goal is impact evaluation—what
about a program has worked and what has not, what has been cost-
effective and what has not, and what aspects can be improved in the
next iteration of the program?
The problem, said Whitty, is that those who ask for evaluations
often are asking for a single evaluation that meets all three goals at
the same time. “If someone asks you for all three, you have to tell
them that they are different things and they are going to have to
pay more and probably have to do it by at least two different
mechanisms.” This discussion has to take place up-front between the
person who would do the evaluation and the person commissioning
an evaluation to avoid wasting time and money on pointless
activities, he added. Another confounding factor is that most of the
large, complex programs are conceived by what Whitty characterized
as “very smart, very politically connected, and very charismatic true
believers.” The resulting political realities have to be considered in
the initial design discussions between funders and evaluators.
On the side of the evaluators, poorly performed evaluations are
often the result of the difficulty of evaluating complex programs.
“What we are talking about here is intrinsically difficult. Many of
these things are really hard to evaluate.… I do not believe there is
such a thing as perfect design for most of the things we are talking
about in this meeting.” He described assessing whether a design is
poorly conceived based on whether he would change his mind about
a program if the evaluation did not provide the answer he expected
or desired. If the evaluation is not “sufficiently strong
methodologically that you are forced to change your mind,” he
stated, then “you probably should not do the evaluation in the first
place. That seems to me to be a common sense test.” Another
problem that he sees on the delivery side is that evaluations of
complex programs require teams comprising individuals with a wide
range of skills, and assembling such multidisciplinary teams is
difficult. How to facilitate the formation of multidisciplinary teams is
“something that we as donors as well as providers need to think
through,” he said.
LESSONS LEARNED FROM THE GLOBAL FUND TO FIGHT

AIDS, TUBERCULOSIS, AND MALARIA
The Global Fund to Fight AIDS, Tuberculosis, and Malaria was

established in 2002 as an international financing mechanism that
would help countries scale up programs that were shown to be
effective in pilot studies. The Technical Evaluation Reference Group
(TERG) is an independent evaluation advisory group accountable to
the Global Fund’s board for conducting an independent evaluation of
the Global Fund’s business model and impact, and in November
2006 the board commissioned an evaluation after the first 5-year
grant cycle. Working together, TERG and the board defined three
study areas that were mutually interdependent and several
overarching questions for each study area.
Another Random Document on
Scribd Without Any Related Topics
convey the cargo to the city. Another deeper and shorter canal has
been made since then.
Capt. Streeter chose the latter plan and speedily got to work at
discharging the tobacco and staves. With two men I worked in the
hold, breaking out and slinging the cargo, and the rest on deck, at
the tackle, hoisted it out.
Though the ship lay alongside the quay, the captain refused
permission to any of the sailors to go on shore in the evening,
saying all they wanted was to get drunk, and the sailors not relishing
this restriction, and thinking they had done enough hard work on
board the "Dublin," took leave of absence on their own account, and
for awhile every morning found two of the crew missing, until only
three or four were left.
In a fortnight the ship was all discharged and ballasted, the captain
went to the city, settled his freight and cleared for Cronstadt. I had
to curb my love of sight-seeing, as my duties would not allow me to
visit the city. In New Diep I saw the Dutch people, the women with
their gold bands hung across their foreheads, and metal head-
dresses ending in front in two little cullenders holding curls, and the
men with their pipes, even the minsters walking to church on
Sunday smoking.
The clean swept pavement and the white walled houses with their
red tiled roofs confirmed the reputation of that people for order and
cleanliness.
The captain upon returning to the vessel shipped some men, and
put to sea, having a final "growl" at the captain of the steamboat
that towed him out, bestowed upon him rather because he was the
last Dutchman he should see for a while, than because of any
special fault in the person. But he had to take Capt. Streeter's
opinion of his countrymen, and to say the least they were not very
flattering to his national pride. "Slower than real estate in Chelsea;"
"don't know enough to go into the house when it rains;" "put two
ideas in their heads and they'd bu'st," were a few of the favorite
phrases made to apply to the subject under consideration, as many
times before they had been applied by Capt. Streeter to such
unfortunate people as came into the world outside of the limits of
"free and enlightened America."
In three days we rounded the north of Denmark and squared away
through the Cattegat with a fresh north-west wind. Before we
reached Elsinore we had a change of wind to the southward, and
were all day beating up the roads, where we anchored at dark. The
next morning we started, with a fleet of one hundred and fifty
vessels, to beat into the Baltic. All hands were on deck, and we
tacked every fifteen minutes. As the Dublin was flying light, and
most of the fleet were coal-laden, she soon distanced them all, and
at sunset we weathered Falsterbo and squared away up the Baltic.
After leaving New Diep, a change came over the captain; the
restraint which he had seemed to impose upon his passions during
this voyage, vanished, and he acted as though intent upon making
up for lost time, and relieving himself of an accumulation of malice
and profanity. In a head wind or calm he would throw his hat on
deck and jump on it, pouring forth abundant curses, and once even
went so far as to shake his fist aloft and swear at "Him who made
the calm." The sailors shook their heads and remarked to each other
that the old man wasn't helping things much, and in the forecastle
they told stories about ships being becalmed until the crew starved,
or until the grass grew so long on her bottom that it took root at the
bottom of the sea and held her fast when at last a breeze came.
The crew behaved pretty well, were very civil and prompt in obeying
orders, and proved themselves good "sailor men" withal.
After the captain had about exhausted his vocabulary on the calm,
he felt the need of something or person else to vent his spite upon;
and as the crew, who usually received these attentions, hardly gave
the excuse in this case, he very suddenly turned upon the second
mate, watched me every moment, and criticised every act that could
be in any way twisted so as to bear it.
He had always appeared more friendly to me than to any one else,
and this sudden change took everyone by surprise. It could hardly
be accounted for except by supposing it to be the expression of his
displeasure at my failure to develop into an officer after his own
heart.
It was soon evident that he had returned to the worst phase of his
last voyage. I, of all others, had occasion to notice it, for the
captain's peculiar attentions were bestowed upon me. His piercing
eye was fastened upon me during the greater part of the day, and
often in the night he crept stealthily on deck in hope of discovering
some neglect of duty, but always found me awake, and the yards
and sails trimmed as they should be, unless it happened that he
came out within a few moments of a little change of wind, and on
one or two such occasions he declared it had been so for half an
hour, and taunted me with inattention, or threw out a hint that he
suspected me of having been asleep—the greatest fault an officer
can be guilty of. His principal reason for the latter suspicion on one
occasion was that he had a dream about wild horses, which never
occurred except when an officer was asleep. He had proved it
several times, and never knew it to fail. Mr. Jones never went to
sleep but once on deck, and that time the captain woke up in the
midst of this dream and caught him.
These things were very galling, but I was able to avoid any
disrespectful response, until one morning his taunts were heaped
upon me beyond endurance, and I had to answer back.
My watch came on deck at 8 a.m., and the captain told me to take a
pull of the main tack. He stood superintending the work as usual,
and as we hauled on the rope he shouted out what were supposed
to be encouraging orders: "Haul, you wicked rascals." "Lay out your
beef on it;—bend your backs to it; you wouldn't haul a mackerel off
a gridiron!" Finally, upon his calling out: "Haul away!" I understood
him to say "belay," and giving that order to the men the rope was
made fast.
"How dare you belay a rope when I'm looking out for it?" shouted
the captain in a rage.
"I thought you ordered me to," said I.
This was an unfortunate speech, as Capt. Streeter had a decided
animosity to anyone's using the word thought.
"What business have you got to think, I'd like to know," he replied.
"You didn't ship for that. I'll make you know your place. I'm the only
man that's allowed to think aboard of this ship. You'll try to take
charge, if I let you keep on with your airs a little longer. You swing
about the decks now as though the ship belonged to you."
These phrases and several others were rattled off, one after the
other, and interlarded plentifully with oaths. Meanwhile I and the
whole watch stood gazing in wonder at the captain, scarcely
knowing what to make of this great ado about nothing. He walked
aft a few steps and turned to watch my movements as I set the men
at work. The mate was standing by the main hatch, and he told me
to let one of my watch sew some canvas on the foot of the mainsail,
and directed me to let him sit in the bight of a main buntline while
he worked at it. I started the man at his job exactly as the mate
wished, but as the man caught hold of the buntline to swing himself
up to the desired position, the captain burst out upon me again:
"What kind of back-handed work is that? Why don't you lower the
man down in a bo's'n's chair? I believe if you got two ideas in your
head it would bu'st. I'd like to know what is the matter with you?"
"The matter is," said I, "that I've always been treated decently till I
came here, and I'm not used to be cursed about and snarled at as if
I was a loblolly boy. Because I'm good natured you're trying to
impose on me, but I can't stand everything."
"If you say another word I'll knock your head off," said Capt.
Streeter, shaking his huge fist in my face. "Don't undertake to dictate
to me what kind of talk I use. I'd swear if the owner and God
Almighty were here." Then he said: "No! I won't fight you, if you
were a man of my size I would, but I'll treat you like a boy that's
beneath my notice that way. But after this I'll keep you in your place.
Go set your men to work, and mind you behave yourself."
That day Capt. Streeter paced the deck a good deal, evidently in
deep thought, and in the evening after supper he called me into the
cabin.
"Mr. A——" said he, "do you know that a man who has had any
education can give a slur that'll hurt a good deal more than another
man can. Now, I feel one word from you more than I do a dozen
from any one else, and I feel hurt at the way you spoke to me this
morning."
"I've always tried to be respectful to you, sir," I replied, "and I think
I've been more so than any body else would have been, because
I've been anxious that no one should think I put on any airs on
account of your familiarity with me. For the last week you've done
nothing but snarl at me and pick upon me. I know, of course, that
I'm at fault sometimes, but not as much as you try to make out."
"You can't expect a sea captain to be as mild as a parson all the
time," said Capt. Streeter. "You must make allowances. If I'm not
quite perfect I want you to respect me as your captain!"
"I always mean to respect you as my captain; but, if you'll allow me
to speak the plain truth, it's impossible to respect you as a man, and
I'm not always able to conceal my private feelings."
"If you can't respect me as a man, I want you to as your captain,"
said Capt. Streeter, biting his lips and looking as though he had
received a slur that cut pretty deep. "That'll do."
Capt. Streeter felt that the account stood rather against him, and
took continual opportunities to annoy me, and occasionally repeated
the sentence which closed his cabin conference, showing that my
remark had taken strong hold upon him.
The night before we reached Cronstadt I had a good talk with the
captain, and he came to the conclusion he had better turn his
attentions to somebody else, and we gradually got to better terms
with each other.
In Cronstadt we discharged ballast and loaded a cargo of iron and
hemp. The crew were called every morning at half past four, which
of course was not very agreeable, and one morning an Irish sailor
growled so much about it, the mate went into the forecastle and
struck him two or three blows with his brass knuckles. A half hour
afterwards the mate picked up a handspike and struck him a blow
across the stomach, and after breakfast he told the captain of it, and
the man was called aft and severely beaten on the back with the
end of the main clew-garnet, a good sized rope. No more complaints
were made about early rising. The mate had a row with another
sailor one day, and receiving some insolence, threw a heavy piece of
wood at his head, which fortunately missed its mark. The man was
so frightened that he deserted that night with one of his shipmates.
The ship was loaded so deeply and was leaking so much, and
moreover had such wretched pumps, that many misgivings were
expressed as to her ever crossing the Atlantic safely. The
superstitious among the crew were still more disaffected when two
Finnish sailors came on board, for a Finn is believed to have dealings
with the evil one, and to be a dangerous shipmate. We sailed for
Boston one September morning, and beat down the Gulf of Finland.
The crew that were shipped at New Diep were to get fifteen dollars
a month, but wages were higher in Cronstadt, and the two Finnish
sailors had shipped for twenty dollars. They had signed articles to
that effect, drawn up by the American Consul. This grieved the
captain's economic soul, and the day after we sailed, he called one
of the Finns into the cabin and summoned me for a witness. He told
the man that if he didn't prove to be a first-class, able seaman, he
should cut his wages down to ten dollars a month; but, if he would
sign the articles that the rest of the crew were on, and accept fifteen
dollars, he would say nothing about his seamanship. The man was
confident of his ability, and had every appearance of a thorough
seaman. He understood English imperfectly, and was somewhat
bewildered by this proposition, but he realized it was a scheme to
defraud him of five dollars a month, and he respectfully declined to
sign the new articles, saying, he had signed once before the consul
and that was his bargain. After a little useless argument, the captain
rose and shut the cabin door; then he caught the man by the neck
with his left hand, and gave him a blow with his right fist that
knocked him down. He jumped on his chest two or three times with
his whole weight; and then kneeling on top of him pounded his face
severely. The man cried out for mercy and promised to sign. He was
then helped to the table and wrote his name on the fifteen dollar
articles. The other Finn was at the wheel at the time, and whether
he heard anything of what was going on or not, he seemed to lose
his head just then, and ran the ship off her course. The mate,
perceiving it, struck him and put another man in his place. He was
just coming forward as the captain and his shipmate stepped out of
the cabin. The bruised face of his comrade startled him, and when
the captain told him to go into the cabin he refused, supposing he
was going to be beaten for his bad steering. The captain, without
further words, seized a belaying pin from the rail and hit him a
powerful blow on the head, which cut a deep gash on the side of his
forehead, and in a moment his face was one mass of blood. The
steward and myself carried him into the cabin, by his head and
heels, and seating him on a stool in a state-room, bound up his
broken head with strips of sail cloth in lieu of rags. The captain
brought a pen to him and told him to write his name on the old
articles.
"What ish dis?" he asked.
"Do as you're told," said the captain, and the man signed.
The captain then put a pair of handcuffs on the man's wrists, though
he was as quiet as possible, and he was left to meditate on the
privileges of sailing under that symbol of freedom and justice, the
American flag.
CHAPTER IX.
THE "DUBLIN"—CONCLUDED.
WHEN eight days out from Cronstadt the ship was in the North Sea
about forty miles S. W. of the Naze of Norway. The weather was
rainy and the sky dark and threatening. The wind gradually
increased to a gale from the westward, and in a few hours the ship
was hove to under the close-reefed main-topsail, laboring heavily in
the ugly confused sea. Careful attention to the pumps showed that
the ship was leaking more than usual, keeping the pumps constantly
going. The sand washed to the well-room and choked the pumps,
which had not force enough to throw it out. It was blowing a living
gale, the ship was leaking badly and the pumps were useless, the
alternative that was presented was to founder at sea, or run for
some port. Accordingly, in the afternoon, the captain ordered the
yards to be squared, and the ship scudded before wind and sea
towards the Cattegat. Getting one pump clear, the crew, by constant
work, kept the water from gaining rapidly; but a new danger was
now before us. The captain had not seen the sun for twenty-four
hours and was not very confident as to the ship's position; she was
running towards the land, and an error of a few miles in the
reckoning might result in the loss of all on board before the next
day-break. The Coast of Denmark is very low and cannot be seen far
at sea even in clear weather, and though Captain Streeter hoped to
fetch to the northward of it, still he felt very much concerned, as the
distance would probably be run before day-light. I shall never forget
my feelings as I stood by the wheel that night, in the middle watch,
while the ship scudded before the howling tempest in the pitchy
darkness, perhaps towards sudden and certain destruction.
The negro steward had the best eyes of any man in the ship, and at
the first dawn of day he went forward and looked anxiously and
earnestly ahead. In a moment the cry, "Land, ho!" sounded, and he
ran aft and reported to the captain that he could see the land not
more than five miles off. Calling all hands the captain hauled the ship
by the wind and crowded all the sail possible in order to keep the
ship off the lee-shore and weather the northern point of Denmark,
which, as the day broke, appeared on the lee bow. He found that he
had missed his course by only eight or ten miles, but had it not been
for the steward's eye-sight, or had day dawned fifteen minutes later,
the error would have proved fatal. The ship struggled bravely against
the gale under double-reefed topsails, a press of canvas that
threatened "to take the sticks out of her," but she seemed inspirited
by the nearness of the dangerous coast, and when she finally
fetched by the Scaw and squared away across the Cattegat, all on
board felt that the question of life and death was decided in their
favor, and cheerfulness and thankfulness took possession of them.
One hundred and fifty vessels and three hundred lives were reported
lost in that gale. That afternoon the ship took a pilot off the Swedish
coast, and proceeding seven miles up the River Gotha, anchored at
Masthugget, a suburb of Gothenburg.
The next day a survey was held and it was decided that the ship
must discharge and be hove down, though the leak was no more
than good pumps could have kept under. Here was a dismal
prospect. It was October and three months must pass before the
"Dublin" could be ready for sea again, and then it would be January,
and probably the river would be frozen over, so that she would be
ice-bound till spring. The dreary looking country and the low
unattractive town which was in sight presented little temptation to a
long residence, and great discussions went on in the forecastle,
whether the "old man" would discharge the crew or not.
By law, sailors discharged from an American ship abroad are entitled
to three months' extra pay, one third of which goes to the Consul.
This made the crew still more anxious to leave and they impatiently
awaited the decision. The Consul, with unusual liberality, told Capt.
Streeter that if his crew wished to be discharged and would at once
take passage in the steamer for England, he might let them go
without any extra pay. The men were a little disappointed at this,
but were so dissatisfied with the ship that they preferred to leave on
these terms, and were accordingly paid off and took the steamer for
Hull, England. As they passed the "Dublin" on their way down the
river they waved their hats and one of them shouted "Bad luck to
the old hooker."
One evening in New Diep, old Harry went into the between decks on
some duty, and as it was rather dark there he fell through the
hatchway into the hold and broke his right arm. Captain Streeter was
too economical to employ a doctor, and too strict a disciplinarian to
allow the sailor to go on shore to see one; he also believed that he
knew more than all the medical fraternity put together. He therefore
set the bone himself, but did it so badly that the arm was nearly
useless after the bones had knit. Harry was very downhearted about
it, for now he could no longer ship as an able seaman and, as usual,
when he was discharged at Gothenburg he sought relief from his
sorrows in drink. In a few days he was picked up in the gutter, one
cold morning, penniless and almost naked. The United States Consul
kindly gave him some clothes, and wished him to accept a very
serviceable coat which had belonged to his coachman. But the sailor
said: "Old Harry is an old man-o'-wars-man and he can't wear a
coachman's coat. Cut those big buttons off and I'll take it." In his
buttonless coat he was put on board the steamer for England and
disappeared.
One Sunday I went up to Gothenburg, in one of the little steam
launches that ply up and down the river. In the evening after my
return I told the mate of my visit; and after hearing my description
of the city, he said:
"I wish you'd been aboard this afternoon when Capt. Mann of the
brig "Hong Kong" was down in the cabin, visiting the old man. They
talked so loud I couldn't help hearing all they said, though I didn't
listen. You know you told me Capt. Streeter never commanded a
vessel before he had this one; but if you had heard his yarns, you'd
have learned that he's had charge of a whole fleet of ships, and he
had such a great reputation that the Emperor of Russia wanted him
to command one of his "cravats," as he called it, but I suppose he
meant corvettes. He told the Emperor that if he took charge of her,
the first thing he'd do would be to run her into action and get the
crew killed off, in order to make room for true-born Americans—the
only men who could get two ideas in their heads without bursting.
The Emperor didn't like this plan of disposing of his subjects very
well, but he knew that Capt. Streeter was such a smart man that he
still urged him to accept, until the affair ended by the captain telling
him he wouldn't take one of his ships if he'd give her to him.
"That wasn't the yarn I started to tell you though. Capt. Streeter
said that he commanded the ship "Seaman's Bride." (I'm pretty sure
he's been second mate of her). He was loading teas at Shanghae on
owners' account, and they wrote to him to make the quickest
passage home he possibly could, and not to spare either spars, sails
or rigging. In eighty-four days after leaving Shanghae he dropped
anchor off the Battery; the quickest passage ever made. He had
carried away a set of top-gallantmasts, sprung the fore topmast,
mainmast-head, and fore and main topsail-yards, and blown away
two suits of sails. On discharging her, they found two of her deck
beams broken, five knees started in the between decks, and four
hanging knees in the lower hold broken, and so on till he ran up a
list longer than a bill of repairs on an underwriter's job. When he got
the ship's damages told, he began on the damages to sailors, and I
tell you they were still worse. He killed two men outright and in New
York nineteen men went ashore with broken heads, all fixed to order
by himself, for his mates didn't know anything, and feared
everything in the shape of sailors, and he had to lick them too. I
never heard a man tell a straighter story in my life. I believe the old
man would beat Tom Pepper at a yarn, and they say he was more
than a match for the Old Nick at lying. I'd like to be behind the door
when the match was going on, anyway. I never thought he told the
truth very hard, but I believe now he's forgotten how, if he ever
knew."
"Why," I added, "you know he said himself one day, 'I never tell the
truth except when a lie won't answer,' and I thought he came nearer
telling the truth than usual when he said that."
"My opinion of Capt. Streeter," said the mate, "is that he would be a
thundering rascal if he dared to be, but he hasn't got the pluck, and
he tries to get the credit of it by making up in lies what he hasn't
courage to do."
"He's a pretty hard man though," said I; "I've seen him handle some
sailors very roughly."
"Hard man," said the mate; "I wish you could have seen the work in
some ships I've been in. What courage does it take for a great two-
fisted fellow like him to handle a single sailor. There was old "Blower
Aiken," who used to keep a bucket full of coal on the poop to heave
at the sailors, and when they were at the main braces, if they didn't
haul hard enough to suit, and it wasn't very often they did, he'd get
up on the after-house, and jump down on top of one of the men,
and then turn to and lick the whole watch. When I was in the
packet-ship "Mountaineer," along with 'Bully Nat Johnson,' I was with
what I call a hard man. If the man at the wheel got the ship a little
off her course, I've seen him pick up a boat-hook and run it through
the man's cheek, and keep him standing at the wheel till his trick
was done.
"We were coming home from Liverpool once, and went out of the
North Channel; but then the wind came from the northward and
blew a living gale. This brought the Irish coast on our lee, and the
'old man' carried sail pretty hard to claw off. Our fore-topsail blew all
to ribbons, and while we were up bending another, a boy fell off the
lee yard arm. Only the man next to him noticed him, and it was
blowing so hard he could not make any one hear to windward. The
mate, who was on deck, saw it, and beckoned to me to come down.
(I was third mate of her). When I got on deck, he told me what had
happened, and he had just thrown a rope which the boy had got
hold of; for the ship of course was only just drifting. We tried to haul
him in, but when he was nearly up, he slipped his hold and fell into
the water. He floated aft, and caught hold of the main chain-plates;
and just as I was going to get a rope round him the old man yelled
out to me: 'Go up on that fore-topsail yard.'"
"'There's a boy overboard, sir,'" said I.
"'I don't care,' said he; 'let him help himself. If we don't get that
fore-topsail bent we'll all be lost.'"
"While I stopped for this talk, a sea had washed the boy away, so I
went up aloft.
"The sailors were so frightened at the force of the gale and flapping
of the remnant of the old sail that they had all laid in off the yard,
and wouldn't go out again. The old man came up and kicked them,
and jumped on their heads as they stood in the rigging and top, but
they wouldn't stir, and at last the second mate and I crawled out on
the weather yard arm, and he lashed me on to the yard with a
gasket, and then I cut away the old sail and hauled out the head of
the new one, for the men came out when they found some one to
take the weather earing.
"Old Johnson is dead now. They called him one of the smartest men
that sailed out of New York, and he could always command his own
wages, but I guess old Jimmy Squarefoot is putting him over the
road now for—four bells! you don't say an hour's gone a'ready; you
must want to turn in."
The cargo was discharged into lighters and stored on shore. Then
the "Dublin" was hauled into the shipyard and her inmates, captain,
two mates, cook and steward moved to boarding houses on shore.
The ship was hove down on her side, caulked and sheathed, and all
day long we stood on the rafts alongside and went through the form
of watching the workmen. We had a vacation from the captain's
society in the evening, except when I had to go up to his lodgings
and write his business letters for him.
The beginning of January found the ship again loaded and ready for
sea, only waiting for sailors, who were very scarce; partly so,
because the captain had told so many fighting stories in the ship-
chandlers' stores that the bad reputation the old crew had given the
vessel had been confirmed and increased. The river froze over, and
though a channel was kept open by steamers, this might be
expected to close any night, and unless the ship desired to remain
until spring it was time for her to leave. So she was towed down to
an outer harbor through seven miles of ice and in a few days the
captain joined her with a crew of young Swedes—no old sailors
being willing to join the vessel. The river was now frozen entirely
over, and even in this harbor ice had formed for two miles out
amongst the islands towards the sea. Accordingly the captain made
a bargain with the fishermen of the place to break out the ship, and
they assembled in force with their ice boats. These were built with
sharp bows which the men lifted and struck upon the ice, and as it
gave way they jumped on to the boat. When a space had been
cleared ahead of the ship, her fore-topsail was set, and she crashed
along until brought up by the firm ice, when the boats again went to
work while the ship held on by a line toggled into the ice astern. In
this way we worked all of one day, and at its sunset found ourselves
in open water; then we made sail and steered to the westward,
delighted to be at last homeward bound.
Ten out of fifteen of the crew could not speak English and most of
them were young men and very poor sailors. But they were "willing"
and well disposed, and the knowledge of Swedish I had acquired in
the long winter evenings enabled me to work ship with them quite
easily.
The captain had not improved his opportunity to master the
language, and it nearly drove him distracted when the men ran to
the wrong end of the ship to execute an order. He tried on his
"tantrums" at first, and issued his volleys of curses and
blackguardism from the top of the after-house, but the sailors only
turned and stared at him with their mouths open in wonder. He gave
this up after awhile and we had quite a peaceable passage.
The ship made good headway in spite of this, and soon again passed
Fair Island and the inevitable boat-load of beggars, and commenced
battling with the Atlantic. Lat. 60° N. in the month of January is not
a very agreeable locality to sail in, and gales of wind were frequent
visitors. We were about half way across the ocean, when one
evening the captain, anxious to finish his passage, was "carrying on"
to the ship with the double-reefed topsails, steering west, with the
wind north. The ship was thus right in the trough of the sea and as
it blew a fresh gale, and the seas were beginning to roll higher and
higher, the men cast many uneasy glances to windward. At a little
before eight o'clock, while the watch was pumping ship, a sea broke
on board forward, and breaking down the bulwarks and tearing
away the water-casks from their lashings, swept aft, and catching up
the crew at the pumps carried them down into the lee scuppers. As I
was washing about with the surges of the water I at first thought I
was overboard, but after awhile managed to get on my feet and was
pleased to find the deck still under them. Half a dozen men were
standing up to their necks in water howling Swedish exclamations,
declaring they had broken arms and legs and so on, and when I got
around to windward the moon broke out through the clouds and
showed a scene of the greatest confusion. Boat, spars, and ropes
had been washed off the house, and the decks were piled up with
the debris of water-casks and bulwarks. As soon as the frightened
sailors could be got to work sail was reduced, and upon surveying
the damage done, it was found that the ship's stem was started
away from the "wood ends" and the water must be pouring into the
hold. Some hands at once were set to work to prepare for throwing
overboard cargo from the fore-hatch and the rest sent to the pumps.
To the great relief of all, the noble pumps that had been furnished in
Gothenburg worked splendidly and freed the ship from water. As
soon as the mate could get time he began to count the men, fearing
that some had been washed overboard, and failing to make out the
number he took a look into the forecastle and discovered five sailors
snugly ensconsed in their bunks. When they spied the mate they all
began to groan and cry, and upon inquiry he learned that one had
broken his back, two had broken legs, and the other two broken
arms. He called the captain, who made a hasty examination which
convinced him they were more frightened than hurt, and he said:
"I'll give you all just five minutes to get out on deck, and if you ai'n't
out in that time I'll come in and drive you out with a handspike."
This threat brought the broken legs and arms into action and they all
made their appearance within the prescribed time, somewhat
bruised but none of them seriously injured.
Two days after this the water was quite smooth and the captain
desired to do something to stop the leak forward, which kept the
pumps going nearly all the time. It was necessary for some one to
go over the bow in a "bow-line," and as the weather was cold and
the person would dip in the water, it was an unpleasant as well as a
dangerous task. The captain disliked to order any one to do it, but
the mate volunteered to the work. A bed-quilt was cut up into long
strips, and being lowered down over the bow in the bight of a rope
with a stick, the mate proceeded to stuff the quilt into the open
seam and then nailed canvas over it down to the water's edge. At
every dip he was nearly submerged in the cold waves, but manfully
did his work until the last nail was driven, and then the hammer
dropped from his benumbed hand and he was drawn on board
thoroughly chilled. He was taken to the cabin and treated to a stiff
drink of whiskey. He soon recovered from the immediate effects of
his exposure, though for some time after he felt the drain it made on
his powers of endurance. The leak was reduced one half by his
labors and he was regarded as a hero.
All felt very anxious upon approaching the coast, fearing to
encounter heavy weather while the ship was in this crippled
condition. But in spite of our hopes the gales were destined to come.
A few days after this, another gale set in at midnight, and at one
o'clock in the morning all hands were called to double-reef the
topsails. It was a dark, wild night, blowing hard with rain and sleet,
and very cold. The crew were so worn out with exposure they were
not very lively, and we were an hour and a quarter before we got
below again. In reefing the fore-topsail we were aloft twenty
minutes, the sail being wet and stiff and the yard not being properly
braced to the wind so as to "spill" the sail. The captain swore we
had been up there two hours, and said he would see if the main-
topsail couldn't be worked quicker. He got his rope's end, and at the
order "lay aloft," he flew around the deck and beat every man into
the rigging; then he followed them aloft, thrashing at every one he
reached. When they laid out on the yard, he walked out to each yard
arm holding on to the top-gallant studding-sail booms, which were
triced up, and beat each man over the head and shoulders. Standing
in the maintop he struck at each sailor as he passed down. We were
five minutes longer than we had been at the fore-topsail, but the
captain flattered himself he had hurried matters. In memory, this
dark night, the fierce storm, the cold blinding sleet, the weak and
disheartened crew and the worst storm of rage, curses and blows
from the captain, form an abiding impression of a demoniacal event.
The ship was so deep her decks were always wet, and seas broke
over her continually when the wind attained to any force. Even if
protected with oil-clothing a sea would often knock one down and
soak him. To stand on deck four hours in a cold, stormy night,
soaked to the skin and with boots saturated and partly filled with
water, makes one's life seem to be oozing away. No wonder sailors
are short-lived; sea exposure and shore degradation soon use them
up!
When about in the longitude of Cape Sable we took a fresh southerly
gale with warm rainy weather, and the same afternoon it suddenly
hauled to the north-west, increasing in force, and the weather
becoming very cold. The wet sails froze so stiff that it was with the
greatest difficulty that the crew could furl them, and while trying to
close-reef the fore-topsail five of the men had their hands frozen and
with difficulty got safely on deck. The "Dublin" had the old-fashioned
whole topsails and it was a dreadful job to handle them. The plan of
making two handy sails out of one large one as in the "Howe's Rig,"
which is now almost universally adopted, is one of the greatest
blessings of the age to the mariner, and yearly saves numbers of
lives and a vast amount of hardship. Some of the men, whose hands
were frozen, restored the circulation by rubbing them in the icy
water which washed over the deck, but two of them were disabled,
and upon arrival in port had to submit to the amputation of some of
the fingers and toes. I had both hands frozen, but soon thawed
them out in the cold sea water.
The ship was now hove to on the starboard tack, the gale was
blowing fiercely, and ice making on the ship. The clothes of the men
were frozen upon them, and when the watch was ordered to go
below I took the last dry clothes I had from my chest and turned
into my bunk. I was only just going to sleep, when above the noise
of the gale sounded the rustling and slatting of the fore-topsail,
which had blown adrift, and then came the mate's cry, "all hands
ahoy! Rouse out here and furl the fore-topsail." This was a moment
of real hardship, and it required a great deal of heroism to spring
from one's bunk and face the freezing gale aloft. I confess I shirked
duty and waited for a second call, which fortunately did not come. A
few sailors soon appeared on deck, and the rest too much terrified
or too irresolute to meet the harsh duty were dragged out of their
bunks by the mate and driven aloft, with threats of blows from a
handspike he carried in his hands.
For fifteen minutes the crew battled with the stiff icy sail. Again and
again they had it gathered up and the blast would sweep it from
their benumbed hands, but finally the gaskets were passed around it
and the order was given to "lay down." The sailors then turned in
and rolled themselves up in their blankets to try one phase of a
sailor's life, "turn in wet and turn out smoking."
All the next day the gale raged with fury, the ice was a foot thick on
deck, and the ropes and rigging were masses of ice. It was
impossible to work ship or make sail and we let her lie and drift to
the southward. The day succeeding, the gale moderated and the
thermometer suddenly started up. Trying the temperature of the
water alongside, we found ourselves in the Gulf Stream. All that day
we were employed drawing up the warm water from alongside,
pouring it over the rigging and beating off the ice. At night we got it
sufficiently cleared to allow us to make sail. Fine weather succeeded
and in a few days we found ourselves on George's Bank. The captain
hailed the fishing schooner, "Eliza A. Proctor," to find out our
position, as he was somewhat distrustful of the accuracy of his
chronometer.
"Schooner ahoy!"
"Halloa," answered a shrill voice.
"What is your longitude?"
"We hai'n't got no longitude; we're after fish!"
"How does George's Shoal bear?"
"Nor' West by North."
As we passed the schooner Capt. Streeter discovered that the
skipper was his mate of two voyages previous—Mr. Foster, whom he
had quarrelled with and discharged from the ship in Mobile. The
bearing he gave did not at all agree with the reckoning; the captain
had some misgivings as to the skipper's information and decided not
to trust to it. The schooner "Emporia" afterwards gave us another
bearing and when we sighted Cape Cod we found Foster had
deceived us, and given a course that would have wrecked the ship if
it had been followed. He evidently did it out of spite to his old
commander.
Capt. Streeter was weather-wise, and continually prophesied the
changes of the wind. Once when it had been blowing from the
north-west for two or three days, it began to moderate and give
evidence that this wind had had its day. The captain said in the
evening: "This wind is about done now, it will haul around to the
eastward, going by the north, or it may die away calm and haul
around by the south." At four in the morning I called him and told
him there was an easterly breeze.
"Which way did it haul?"
"By the north, sir."
"Didn't I tell you so?" said the captain.
The evening we made Cape Cod, the sky began to clear in the
westward and a light breeze came from that direction. "Now," said
the captain, "we are in for it. It's just my luck. It's going to blow a
living gale of wind from the nor'west; we shall be driven off the
coast and not fetch back here for a fortnight." This was rather
disheartening and I couldn't help replying: "If I thought so I
wouldn't say it, for I think we have had quite trouble enough without
borrowing any."
"I tell you what it is young man," said the captain, "there's a
difference between borrowing trouble and being weather-wise."
A light westerly breeze blew all the next day. We beat up the bay
with fine weather and off Boston light took the steam-tug "R.B.
Forbes," which towed us quickly up to Lewis' Wharf, just as the day
had ended. I heard a familiar voice through the darkness, and the
ship was no sooner fast, than I went over the rail and for two days
abandoning myself to the joys of home I tried to forget that there
ever was a "Ship Dublin." Then I visited the ship, and the captain at
once inquired "Did you notice how it blew last night? I knew it was
coming; I'm not often deceived about the weather." I received the
compliment of being asked to make another voyage in the ship and
the black eyes snapped at my rather peremptory refusal.
The owner was offended with me for leaving, and finding that I had
suffered in his good opinion by doing so, I ventured after some days,
to excuse myself by saying just enough about Capt. Streeter to
justify my conduct. He was very indignant, wouldn't allow such a
man to sail for him, but didn't see how he could discharge him just
then. He would tell him to do differently though. One of the partners
remarked, "Oh, they all swear and fight, and Capt. Streeter is the
smartest commander we have ever had."
He sailed again on another Russia voyage with the old instructions to
"use his best efforts to suppress all vice and immorality on board
and promote the welfare of his crew." How he did it the following
letter from Mr. Wright the mate, written from Cronstadt will tell:
"I suppose you would like to hear how this old boat gets along and
what kind of a voyage we've had so far. When we left the wharf at
Boston, I called the men to come out of the forecastle and go to
work, but the answers I got weren't very polite. They called out, 'we
want to make our beds up; don't get your temper in an uproar; don't
fret' and some other remarks that you can fancy. I got pretty mad,
and I just picked up an iron belaying-pin and went into the
forecastle and made Rome howl. All hands turned on me, and I had
all the fighting I wanted. Things got too hot for me and I had to go
aft and ask the old man to come in and help me. I thought he would
be very glad to have such a chance for "spiflicating sailors" as he
calls it, and some men I've sailed with wouldn't have asked for
better sport than to walk into those sailors and make them take the
measure for their coffins on deck. The way the old man showed the
white feather surprised me. He got on top of the after-house, with a
pistol in his hand, and called to the men to come aft, and talked to
them as mild as a sucking parson. They were pretty sullen, and five
men swore they wouldn't do a hands turn of work on board. The old
man told me to put them in irons, and I did it without much trouble,
for they had too much headache to make any more fuss. I put a
rope through between their wrists and triced them up with the main
lift tackles till their feet only just touched the deck. It wasn't long
before they wanted to turn to. One was hurt so bad that we thought
he would die, and he has been laid up the whole passage. I have
had several sprees with the men since, but now I only have to hold
up my finger and they mind me. The old man doesn't say much to
the sailors, but he's down on the second mate, who is a youngster,
and doesn't know much, and he hazes him when he wants to let off
steam. We are loading for Boston, and I hope we will get there
soon, for I've been about long enough in the "Dublin." I hope you'll
get a good ship and a captain that'll suit you, but they are scarce
fish to find."
The ship was sold on her return, and the captain entered another
employ. His vessel finally went to Australia. When riding horseback
he was thrown, and broke his ankle. The doctors declared
amputation was necessary. With his usual contempt for medical
opinions he drove them away, and thought he could apply to his own
case the skill he had exercised on Old Harry, but in a few days he
died of lockjaw. His end appeared significant to those who knew
how his powers of speech had been misused.
CHAPTER X.
COAST OF MADAGASCAR.
"A strong nor'wester's blowing, Bill;
Hark! don't ye hear it roar now?
Lord help 'em, how I pities them
Unhappy folks on shore now!"—Wm.
Pitt.
"In noble minds some dregs remain.

Not yet purged off, of spleen and sour
disdain."—Pope.
WE sighted the coast of Madagascar about Fort Dauphin, but at a
distance of nearly fifty miles, so that our view of it was not very
distinct. But it is always a pleasure to a sailor to see land, and a
great relief after many days of wearisome, changing, wave scenery
to let the eye rest upon the everlasting hills. The mountains
symbolize stability and unchangeableness, and as the aspirations of
the heart are ever towards those things most in contrast with
present experience, so the sailor's life of unrest and variableness
disposes his dream of happiness to be one of enduring repose. This
idea the land as contrasted with water expresses to his mind. A
thrilling sensation always accompanies the sound of "Land, ho!" and
longing glances are directed to the faintest loom of land in the
horizon.
Our easterly winds continued and took us to the coast of Africa,
which we sighted about Algoa Bay. Then the wind became light and
variable and with smooth water we sailed slowly along in sight of the
land for four days. One evening we sighted the light on Cape
Agulhas, the south-east point of Africa, and with the wind freshening
at N.N.E. soon ran it out of sight on the starboard quarter. Before
this breeze sprang up we were in company with a large ship showing
Dutch colors. She sailed alongside of us for a considerable time, then
slowly gained ahead, crossed our bow and in a few minutes
disappeared from sight, though the twilight gave a long range to the
vision. Many were the conjectures about her, and some of our crew
will always believe they saw the veritable "Flying Dutchman," the
phantom ship that is supposed to cruise off the Cape of Good Hope.
The legend concerning her, as many will remember, is, that a Dutch
captain, who had encountered very severe gales, was advised to put
the ship back to a port of distress, but swearing a terrible oath he
declared he would beat around the Cape if it took him till the Day of
Judgment. In punishment for his sin he is doomed to battle with the
elements until that day, and his battered hulk, with threadbare sails
and skeleton crew haunts the southern sea.
At eight in the evening we passed another ship bound the same way,
but under close-reefed topsails, whereas we had the royals set. We
hailed her and found she was the "Meteor" from Batavia, but whither
she was bound, or where she hailed from, we failed to learn, for we
went by her so fast there was no time for further questions.
"What can she be doing under that sail?" asked the mate.
"O, she has got a prudent captain," I replied. "The barometer has
been down low for the last two days and no doubt there is a gale of
wind coming on. I can't take in sail though in this country, with a fair
wind blowing, until I see the breeze coming. Every mile here is
precious and as long as there is a chance to gain ahead we must use
it. But you must keep your eyes peeled to-night for nor'west
squalls."
At ten o'clock I carefully inspected the wind and weather. It was a
bright starlight night, with not a cloud to be seen, except that ahead
in the horizon was a low streak that looked like a fog bank. A fresh
breeze was blowing from the northward driving the ship along nine
knots, with the yards just clear of the backstays, all sail being set.
The mate had the watch on deck, and I said to him, "This is fine, I
only hope it will last, but the glass says, No. However, very likely
we'll have good warning before the change comes. Keep a sharp
lookout and if it breezes on, or the weather looks threatening, get
the light sails off of her and give me a call."
In half an hour after that, the mate shouted "clew up the fore royal,"
but no sooner had they let go the halyards than a furious blast from
the north-west struck her flat aback. The helm was put hard up and
having a good deal of headway the vessel fortunately "fell off." None
of the watch below needed a call for every one was out of his bunk
in a few seconds as the bark, nearly on her beam ends, and the
shaking sails, gave their own summons. I was on deck promptly and
shouted, "Lower down the spanker;" but the mate had his men
forward hauling down the jib and flying-jib, for the sheets had
parted and the sails were blowing into ribbons. The second mate got
his watch along aft as soon as possible, and in the meantime I
jumped on top of the after-house to let go the spanker throat-
halyards. As I passed forward of the mizzen-mast to go to leeward,
the wind and the inclination of the vessel gave me a slide, and away
I went head foremost off the house on to the main deck. I had on
rubber boots with my pants tucked into them, and as I fell the
belaying-pin of the main brace went up the right boot leg and there
I hung, heels up and head down in the lee-scuppers, while the good
bark was lying beam on to the hurricane, which threatened every
moment to dismast her, and in the meantime was blowing to pieces
a number of her sails. The night was pitchy dark and the rain poured
in large drops which, with the force of the wind, struck like hail,
while the storm roared with a sound such as that with which the
express train affrights one who stands on the platform of a country
station past which it flies. I managed to extricate myself from this
awkward position, and crawling to windward renewed the directions
for shortening sail. The vessel was run off to the S.E. for two hours
while we took in and furled every sail except the close-reefed main-
topsail and then she was brought to the wind on the port tack.
That was a night we long remembered, and a hard time the crew
had furling the wet sails in the cold rain, but there was one
alleviation to their discomfort, for I had the cook "roused out" and
ordered him to make coffee for all hands; and as soon as she was
hove to, a mug of hot coffee and a cake of hard bread gave them
one of the greatest treats they ever had in their lives.
"The prudent captain got the best of it this time," I said to the mate.
"I'm not so sure of that, sir," said he; "if he's been waiting two days
for the wind to blow we've gained enough distance on him to pay for
a good deal more damage than we've got."
"But it's a lucky job we did not lose our masts," I said; "if there had
been a flaw anywhere they would have gone. Things held on well.
Didn't it give you any warning?"
"No, sir," said the mate. "That bank that was hanging there ahead,
when you were on deck, was what did the mischief. It seemed to
hold about so and didn't look very threatening, but in five minutes it
spread right up over the sky. I made a start to get sail in before it
struck her, but I wasn't in time."
The gale blew very hard through the night and continued for seven
days, but it moderated at times so that we set the whole topsails for
a few hours. Four different times we were obliged to heave to under
the close-reefed main-topsail and once it was "goose-winged." This
time it blew a fearful gale. There was a black overcast sky, hanging
so low down that it seemed not far above the mast heads, and
driving across with great rapidity. Hard hail-squalls now and then
passed over, and every face had to be shielded from the stinging
violence of the hailstones. The sea was tremendous. At times there
would be but one wave in sight, that, the whole ocean, and towering
high up above the rail almost even with the tops it would come
rolling on seeming to bear inevitable destruction; but as it
approached, the good bark would gradually mount up its side, and
then be whirled up and lifted over its summit like a little toy. As the
waves broke, the wind lifted the whole crest into its arms and bore it
onward mingling sea and air, driving the spray in horizontal lines
high aloft across the ship. At about two o'clock in the afternoon a
sea broke alongside and a good portion of its top came tumbling in
over the weather rail. Nothing could resist its force. In went the
galley and forecastle doors, the water-cask lashings gave way, the
pig-pen on the main hatch was smashed all to pieces, the spare
main-yard broke adrift, and the sea, having spent its force, found a
passage for itself through the lee ports.
Fishing off the Cape.

After this gale a calm prevailed for a few hours and we heeded
Horsburgh's praise of the fishing on the Banks of Agulhas, by trying
our fortune with the line. The only result, however, was the
accompanying sketch of the performance.
By these gales we lost eight days on our passage and only gained
one hundred miles in nine days, an inspection of our track for ten
days will show how hard it is sometimes for sailing vessels to make
quick passages.
When fifty days out we sighted the revolving light on Cape of Good
Hope, and the next day having a light westerly wind we stood along
the coast to the northward and enjoyed a fine view of Table
Mountain.
This turning of the corner was a joyous event. Now we pointed the
ship's head towards home and realized that we were actually bound
there, which it was hard to do while our course had any southing in
it. Fine weather regions lay before us, and an immense load was
removed from the captain's mind by the safe "doubling of the Cape."
South-east winds set in next, and we went "rolling down to St.
Helena" before fresh trades, with very fine weather.
The steady winds and settled weather of the South Atlantic are
always taken advantage of by the homeward bound ships to tar-
down the rigging, paint and "fix up" generally for port. It is
customary to keep all hands then, even in ships where it is not the
continual practice. That is, instead of having only half of the crew at
work at a time and alternating every four hours, all hands are kept
on deck in the afternoon from one o'clock until six. They all get
dinner together at twelve and no work is done from noon until one.
At one all "turn to," and either all hands get supper together at six,
or one watch gets theirs at half-past five and the other at six. Under
the watch and watch system a sailor is on deck ten hours out of the
twenty-four on one day and fourteen hours on the next, making
twenty-four hours of work and twenty-four of rest in forty-eight. In
the all-hands system a man is on deck thirteen hours one day and
fourteen and a half the next, making twenty-seven and a half hours
of work and twenty and a half of rest in forty-eight. To the advocates
of the eight-hour system, this may seem an undue proportion of
working hours, but it is to be remembered, however, that half of
these hours occur in the night time, when, if the wind is steady and
weather fine, there is no work to be done, and if the helmsman and
the lookout are wide awake and the crew answer promptly to a
summons, it would not be noticed in most ships if the men stole a
nap on deck between times. But in "hard ships" the men are always
kept moving. The officers of course at all times in their watch on
deck must be wide awake and, it is presumed, on their feet, so that
keeping all hands is more of a privation to them than it is to the
sailors.
The mate asked me one night after we got past the Cape, if he
should begin now to keep all hands until the work was done.
"Do you think you could get the work done with watch and watch?"
"I suppose we could," said the mate, "but we shall have to keep
driving at it right up to Boston Light."
"Did you ever go through the trades with watch and watch?" I
asked.
"No, sir, I never did," said the mate.
"Well, I never did myself till last voyage, then I was so well pleased
with the result that I should like to try it again. The voyage before
that, I came on deck one night, while we were keeping all hands,
and found the second mate sitting on the bumpkin, his arms on the
rail and his head buried down in them, while he was snoring after
the style called 'driving the pigs to market.' The next day I had a talk
with him about his neglect of duty. He acknowledged his fault, but
said it occurred in spite of all he could do. He said he had tried every
way he could think of to keep himself awake. He had walked the
deck until he was compelled from sheer exhaustion to sit down, for
it was a hot sultry night, and he had been on his legs all day long.
He assured me very earnestly that he had not neglected his duty
intentionally. Said he: 'Cap'n, did you ever have any fault to find with
the way I kept the night watch before we had all hands?'
"'No,' I answered, 'none whatever.'
"'No, sir,' he said, 'and I am sure you had no occasion to. Excuse
me,' said he, 'I don't mean to growl at your way of doing things, but
I can't feel that an officer is greatly to blame if he is drowsy at night
in an all-hands ship. I was studying it all out last night while I was
dragging myself fore and aft the deck trying to keep awake, and this
was what I made out: I get nineteen hours to myself out of forty-
eight, and when you take out meal-times, dog-watch, a little time for
keeping clothes in order and what time I give up to the ship in my
watch below to help things along, I can't get more than twelve
hours' sleep in two days. Six hours a day is thought a small
allowance on shore where a man can sleep it right through. But our
rest is so broken I don't believe it does as much good. Three hours
and a half or four hours is the longest sleep one can get at a time,
and then he has to stand four hours on deck before he has a chance
to get another cat nap.'
"I felt he had a good deal of truth on his side, though I didn't like to
tell him so, and I thought a good deal about it afterwards. The next
voyage I resolved to try how watch and watch would work, and
when we got into the S.E. trades, homeward bound, I told the mate
to say to the men: 'There's just so much work that's got to be done
before this ship reaches port; now if you can do it with watch and
watch, you shall have it, but if there's any 'sogering' or loafing you'll
be kept up in the afternoon.'
"We began it. The men all worked with a will, and I am certain that
as much was done as on the previous voyage. I took special pains to
compare, and all through I noticed that there was more drive, and
less loafing, going for a drink, turns round the foremast, and long
spells at the grindstone. On some of the large jobs, too, I had a
good chance to judge. I suppose there's no job that admits of as
many 'soger moves' as scrubbing ship outside. The men come up on
deck every little while to haul up or 'fleet' the stages they are
working on, and then they spin out the time before they get back by
sharpening knives and scrapers, or getting a drink, and a good many
other moves that every one knows, who has ever had to follow up
old sailors.

Evaluation Design For Complex Global Initiatives Workshop Summary 1st Edition Institute of Medicine All Chapters Instant Download

Uploaded by

Copyright:

Available Formats

Evaluation Design For Complex Global Initiatives Workshop Summary 1st Edition Institute of Medicine All Chapters Instant Download

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Evaluation Design For Complex Global Initiatives Workshop Summary 1st Edition Institute of Medicine All Chapters Instant Download

Uploaded by

Copyright:

Available Formats

Full download ebooks at ebookmeta.

Evaluation Design for Complex Global Initiatives

Download more ebook from https://ebookmeta.com

Collaboration Between Health Care and Public Health

Spread Scale and Sustainability in Population Health

Global Health Risk Framework Governance for Global

Financial Incentives to Encourage Development of

Building Health Workforce Capacity Through Community

Cross Sector Responses to Obesity Models for Change

Exploring Shared Value in Global Health and Safety

Preventing Intimate Partner Violence in Uganda Kenya

NOTICE: The workshop that is the subject of this workshop

This activity was supported by contracts between the National

International Standard Book Number-13: 978-0-309-30258-6

Additional copies of this workshop summary are available for sale

Copyright 2014 by the National Academy of Sciences. All rights

Printed in the United States of America

Suggested citation: IOM (Institute of Medicine). 2014. Evaluation

The National Academy of Sciences is a private, nonprofit, self-

The National Academy of Engineering was established in 1964,

The Institute of Medicine was established in 1970 by the National

ANN KURTH (Chair), New York University, New York, NY

This workshop summary has been reviewed in draft form by

RICHARD BERK, University of Pennsylvania

Although the reviewers listed above have provided many

The planning committee and project staff deeply appreciate many

1 INTRODUCTION AND OVERVIEW OF THE WORKSHOP

2 OVERVIEW FRAMEWORK FOR COMPLEX EVALUATIONS

3 FRAMING THE EVALUATION

4 DEVELOPING THE EVALUATION DESIGN AND SELECTING

6 APPLYING QUALITATIVE METHODS TO EVALUATION ON A

7 APPLYING QUANTITATIVE METHODS TO EVALUATION ON A

8 ANALYSIS THROUGH TRIANGULATION AND SYNTHESIS TO

9 EVOLVING METHODS IN EVALUATION SCIENCE

10 LESSONS FROM LARGE-SCALE PROGRAM EVALUATION ON A

11 USING EVALUATION FINDINGS AND COMMUNICATING KEY

12 ENVISIONING A FUTURE FOR EVALUATION

Introduction and Overview of the Workshop1

Every year, public and private funders spend many billions of

TERMS AND OBJECTIVES

In her opening remarks at the workshop, Ann Kurth, professor of

Large-scale—The total cumulative budgets over multiple

Encompassing multiple components, such as varied types of

Evaluations of four large-scale, complex, multi-national health

Global Fund to Fight AIDS, Tuberculosis, and Malaria (GF)

Appendix C provides a comparison of the evaluations for these

MESSAGES FROM THE WORKSHOP

In the final session of the workshop, some of the important

1. Sanjeev Sridharan, director of the Evaluation Centre for

The workshop then closed with reflections from representatives of

What Evaluations Can Do

Governance and Evaluator Independence

Data Availability and Quality

Final Reflections on Future Large-Scale, Complex Evaluations

Document the stories of evaluations.

ORGANIZATION OF THE WORKSHOP REPORT

Following this introductory chapter, Chapter 2 of this summary of

Overview Framework for Complex Evaluations

Important Points Made by the Speaker