See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/266028337

Effective Coverage: A Metric for Monitoring Universal Health Coverage

Article  in  PLoS Medicine · September 2014

DOI: 10.1371/journal.pmed.1001730 · Source: PubMed

CITATIONS READS

129 891

6 authors, including:

Marie Ng Nancy Fullman

University of Washington Seattle University of Washington Seattle
118 PUBLICATIONS   29,993 CITATIONS    204 PUBLICATIONS   35,841 CITATIONS   

SEE PROFILE SEE PROFILE

Joseph L Dieleman
University of Washington Seattle
113 PUBLICATIONS   3,612 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

http://www.healthdata.org/ View project

The state of health in the former republics of the USSR, 1990-2013: analysis of the burden of disease, injuries, and risk factors View project

All content following this page was uploaded by Nancy Fullman on 01 April 2015.

The user has requested enhancement of the downloaded file.

Collection Review

Effective Coverage: A Metric for Monitoring Universal

Health Coverage
Marie Ng, Nancy Fullman, Joseph L. Dieleman, Abraham D. Flaxman, Christopher J. L. Murray,
Stephen S. Lim*
Institute for Health Metrics and Evaluation (IHME), University of Washington, Seattle, Washington, United States of America

metrics for tracking progress. Nevertheless, the question remains: how

Abstract: A major challenge in monitoring universal should progress towards UHC be monitored?
health coverage (UHC) is identifying an indicator that can As noted in a recent World Bank report on UHC [4], in order
adequately capture the multiple components underlying to adequately capture the spectrum of health services represented
the UHC initiative. Effective coverage, which unites by UHC, ‘‘a more holistic approach to the dimensions of access
individual and intervention characteristics into a single needs to be understood.’’ In other words, the most useful metric
metric, offers a direct and flexible means to measure
for monitoring progress in UHC should encompass the multifac-
health system performance at different levels. We view
eted nature of UHC. The monitoring framework put forth by
effective coverage as a relevant and actionable metric for
tracking progress towards achieving UHC. In this paper, WHO and the World Bank Group in 2013 highlighted two major
we review the concept of effective coverage and components critical to assessing UHC progress, namely, service
delineate the three components of the metric — need, coverage and financial protection coverage for all people [5]. For
use, and quality — using several examples. Further, we measuring service coverage, the concept of effective coverage was
explain how the metric can be used for monitoring noted. In contrast to crude coverage, which focuses solely on
interventions at both local and global levels. We also intervention access or use, effective coverage is a measure that
discuss the ways that current health information systems unites intervention need, use, and quality. The comprehensiveness
can support generating estimates of effective coverage. of this metric makes it more suitable for monitoring UHC [6–9].
We conclude by recognizing some of the challenges In this paper, we will review the concept of effective coverage and
associated with producing estimates of effective cover- discuss a number of key issues related to its measurement.
age. Despite these challenges, effective coverage is a
powerful metric that can provide a more nuanced What Is Effective Coverage?
understanding of whether, and how well, a health system
is delivering services to its populations. Effective coverage is defined as the fraction of potential health
gain that is actually delivered to the population through the health
system, given its capacity. It is comprised of three components,
namely, need, use, and quality. Need refers to the individual/
population in need of a particular service; use refers to the use of
This paper is part of the PLOS Universal Health Coverage
services; and quality refers to the actual health benefit experienced
Collection.
from the service. Measuring effective coverage is a significant
advancement over the usual approach of measuring crude
coverage, which only captures access conditional on need. In
Introduction particular, given that use of service alone does not imply that the
Strengthening health systems, ensuring affordability of care,
improving access to quality services, and building capacity are core Citation: Ng M, Fullman N, Dieleman JL, Flaxman AD, Murray CJL,
tenets of universal health coverage (UHC). In 2010, the World Health et al. (2014) Effective Coverage: A Metric for Monitoring Universal Health
Organization (WHO) called for concerted efforts to achieve UHC, Coverage. PLoS Med 11(9): e1001730. doi:10.1371/journal.pmed.1001730
with reducing disparities and promoting opportunities for obtaining Published September 22, 2014
quality care with financial protection as WHO’s underlying goals [1].
Copyright: ß 2014 Ng et al. This is an open-access article distributed under the
The ideology of UHC laid out by WHO was viewed as ambitious and terms of the Creative Commons Attribution License, which permits unrestricted
noble; however, it has been criticized for the lack of specificity for use, distribution, and reproduction in any medium, provided the original author
defining milestones crucial for monitoring progress [2]. The and source are credited.
importance of systematically tracking the progress in attaining Funding: No specific funding was received for this study.
UHC was highlighted in the 2013 World Health Report [3], which Competing Interests: SL is a member of the Editorial Board of PLOS Medicine.
drew attention to the dearth of empirical evidence for assessing and Abbreviations: ANC, antenatal care; ART, antiretroviral therapy; COPD, chronic
informing policies related to UHC. The report identified several key obstructive pulmonary disease; DHS, Demographic and Health Survey; DOTS,
research priorities, which included deepening the understanding of directly observed treatment, short-course; DSM, Diagnostic and Statistical Manual
disease burden at the country level and identifying policy-relevant for Mental Disorders; GA-BW, gestational age and birth weight–adjusted; GBD
2010, Global Burden of Disease 2010 study; HAART, highly active antiretroviral
therapy; IPTp, intermittent preventive therapy during pregnancy; ITN, insecticide-
treated net; IV, instrumental variable; MMR, measles-mumps-rubella; SD,
Collection Review articles synthesize in narrative form the best available Symptomatic Diagnosis; STDs, sexually transmitted diseases; UHC, universal
evidence on a topic. Submission of Collection Review articles is by invitation only, health coverage; VA, verbal autopsy; WHO, World Health Organization.
and they are only published as part of a PLOS Collection as agreed in advance by
the PLOS Medicine Editors. * Email: [email protected]
Provenance: Not commissioned; part of a Collection; externally peer reviewed

PLOS Medicine | www.plosmedicine.org 1 September 2014 | Volume 11 | Issue 9 | e1001730

 Summary Points Box 1. Formal Definition of Effective Coverage
N Effective coverage unites intervention need, use, and At the individual level, effective coverage is defined as the
quality into a simple but data-rich metric, reflecting the fraction of potential health gain that is actually delivered
core components of UHC. to the population through the health system, given its
N Effective coverage can be applied to understand the capacity. The formal definition is as follows:
health gains delivered by interventions at a range of
levels, from individual benefits to national impact. ECij ~(Qij Uij DNij ~1)
N Effective coverage can be measured and used across
resource settings. Lower-income countries can harness where ECij is the effective coverage of individual i with
data from existing survey data to feed into effective intervention j; Qij is the expected quality of intervention j
coverage estimations. as delivered to person i; Uij is the probability of individual i
N The broader use of effective coverage remains hindered receiving intervention j; and Nij is an indicator of whether
individual i is in need of intervention j.
by the availability and quality of health data, especially at
subnational levels. At the population level, effective coverage for a given
intervention is an aggregate of each individual’s probabil-
ity of effective coverage. To estimate effective coverage for
a specific intervention j at the population level, individual-
full benefit of the service is being realized, it is crucial for a health level effective coverage is aggregated as follows:
performance metric to capture not only coverage but also quality.
The calculation of effective coverage is summarized in Box 1. Xn
In addition to capturing quality, effective coverage has another i~1
ECij HGij Pij
unique strength: it is a very flexible metric that can easily be adapted ECj ~ X n ,
HCij Pij
for different contexts and assessed at different administrative levels. i~1
Specifically, effective coverage can be measured for one single
where HGij is the expected health gain from the
intervention and provide information on specific intervention roll-
intervention and Pij is the probability of an individual
out. To the degree that data are available, effective coverage can
needing the intervention.
also be aggregated across a large, diverse set of interventions and To estimate overall effective coverage for the health
proxy the effectiveness of an entire health system. In other words, system of a country, the effective coverage for a set of
effective coverage can be adapted in a manner that reflects country interventions is further aggregated as follows:
needs and health priorities, hence serving as an appropriate
indicator for tracking progress and benchmarking performance. XJ
Effective coverage also can be estimated for subnational levels or
j~1
ECj HGj DKmax
population subgroups, which helps to pinpoint the geographic areas EC~ XJ ,
or populations lagging behind in the receipt of effective interven- j~1
HGj DKmax
tions. Obtaining subnational-level estimates is particularly impor-
tant when effective coverage is used to monitor progress towards where actual expected health gain is conditioned on the
UHC, as UHC has a noteworthy equity agenda and subnational maximum performance Kmax.
estimates are of the utmost importance.
To illustrate how effective coverage has been applied in the real UHC, a country has to consider several factors. First, a country
world, the experience of Mexico is one of the most comprehensive must identify its overall health needs and priorities. Second, a
examples. In an effort to benchmark progress towards improved country has to develop specific strategies for collecting data on
health services and evaluate the impact of its country-wide health need, use, and quality of selected interventions. Third, a country
reforms, the Mexico Ministry of Health adopted effective coverage has to devote resources to enhance both national and subnational
metrics in 2001. A set of interventions reflective of the priority capacity to collect and monitor health information.
health needs were included in the metric, and effective coverage
estimates were derived for each state [10]. The results enable the
Ministry to pinpoint gaps in intervention access and places where Identify Priorities for Effective Coverage Indicators
intervention effectiveness was sub-optimal. Figure 1 shows how Given the broad range of health services delivered by health
effective coverage can reveal gaps in access and inadequacy in systems today, measuring effective coverage for every intervention
service delivery. Specifically, the discrepancy in crude and effective would be impossible. When effective coverage was used to
coverage of hypertension treatment across states shows that in benchmark state-level health system performance in Mexico
some areas the intervention delivered might not have achieved the [10], the Ministry of Health selected a subset of interventions
desired health outcome despite a high level of access. that most directly aligned with the country’s health priorities.
The estimation of effective coverage, however, can be challenging These selected interventions included a mixture of maternal and
because some of the key components can be difficult to measure child health interventions (such as immunizations, antenatal care,
directly and data quality can vary. In the following section, we and skilled birth attendance) and interventions for non-commu-
discuss factors that can affect the collection and use of effective nicable diseases (such as cancer screening and hypertension
coverage in applied settings. treatment). While these selected interventions were far from
exhaustive, they appropriately reflected Mexico’s health needs and
Practical Considerations in Applying Effective priorities throughout the country.
Health need serves as one of the guiding principles in
Coverage
determining what should be prioritized and included for
In order to optimally use effective coverage as a metric for estimating and tracking effective coverage. Individual countries’
monitoring healthy system improvement or progress towards health needs are likely to vary, but some similarities can be found

PLOS Medicine | www.plosmedicine.org 2 September 2014 | Volume 11 | Issue 9 | e1001730

Figure 1. Crude and effective coverage of hypertension treatment across Mexican states, 2005–2006.
doi:10.1371/journal.pmed.1001730.g001

across income levels, geographic location, and cultures. The Global Considerable differences exist across and within world regions,
Burden of Disease 2010 (GBD 2010) study helped to illuminate but a number of commonly experienced disease and injury
some of these trends in health needs [11]. For instance, among burdens exist among subsets of countries. For instance, national
higher income countries, non-communicable diseases compose disease burden studies conducted in the United States, the United
most of their health burdens and corresponding needs. Countries Kingdom, and China identify ischemic heart disease, chronic
that have transitioned from lower to higher levels of income often obstructive pulmonary disease (COPD), stroke, and lung cancer as
experience a parallel transition in health needs, largely shifting from the leading causes of premature mortality and disability for all
disease burdens caused by communicable diseases to those caused three countries [15–17]. This finding implies that some diseases
by non-communicable conditions. Lower-income countries still could be treated as regional, and potentially global, health needs
experience the largest health burdens from infectious diseases and for monitoring effective coverage.
maternal and child conditions, but many of them have documented Identifying a country’s health needs and corresponding
gradually rising rates of injury and more chronic ailments. interventions to address them is a necessary consideration, but it
This diversity of disease burdens across countries implies that is not sufficient. It is also critical to consider the cost-effectiveness
what comprises UHC is likely to vary across settings. For instance, and sustainability of a given intervention or set of interventions
among lower-income countries, UHC may focus around achieving within the health system delivering them. The interventions
basic healthcare for all populations and prioritizing access to selected for tracking effective coverage should align with country-
interventions that address infectious diseases and maternal and specific health needs and a country’s financial and administrative
child health conditions [12]. Therefore, the interventions included capacity to support their provision over time. Efforts have been
in estimating effective coverage would align with these health need made to compile data on optimal intervention delivery options.
priorities, such as antenatal care, skilled birth attendance, and This body of work includes WHO’s Choosing Interventions that
critical surgical procedures [13]. For higher-income countries, are Cost Effective (WHO-CHOICE) [18] and the ongoing
UHC is likely to primarily focus on improved access to treatment projects within the Disease Control Priorities framework [19].
of and preventive services for non-communicable diseases [14]. Through these projects, information on intervention costs and
Subsequently, the interventions included for effective coverage effectiveness has been generated and then assembled such that
estimation for these settings would need to be related to managing comparisons can easily be made across intervention packages and
chronic conditions. Data visualization tools can help identify delivery options. However, these kinds of data are generally only
country-specific disease burdens and health needs: http://www. available at more macro-levels, and have yet to be systematically
healthdata.org/results/data-visualizations. produced at the country level.

PLOS Medicine | www.plosmedicine.org 3 September 2014 | Volume 11 | Issue 9 | e1001730

 When defining key interventions to track for effective coverage, antiretroviral therapy (ART) when they meet specific CD4 count
it is also important to recognize that priorities of health care and thresholds. Individuals who meet diagnostic criteria for depression,
intervention are not solely driven by the disease burden and cost- as set forth by the latest version of the Diagnostic and Statistical
effectiveness. Other considerations, including equity, fairness, Manual of Mental Disorders (DSM), would be the population in
individual rights, and historic and cultural concerns, all play a need of antidepressants, therapy, or a combination of treatments.
significant role in shaping health system and measurement National surveys often collect information on self-reported health
agendas [20]. For instance, some health interventions may be state, which can provide some insights into population-level health
widely delivered to a population to address social demands and needs. However, many types of health conditions may be less
inequity — even if these interventions do not align well with the reliably captured through self-report or may be prone reporting
population’s health needs or the knowledge base of intervention biases (e.g., estimating one’s weight and height or self-reported
cost-effectiveness [21]. HIV status) [23]. Using biological markers provides more accurate
measures of various health conditions [24], and many national
Tracking Intervention Need, Use, and Quality surveys regularly collect data on biological markers throughout the
To estimate effective coverage, the metric’s three components world. Examples include the National Health and Nutrition
— intervention need, use, and quality — need to be measured in a Examination in the United States, China’s Chronic Disease Risk
consistent way. Data on a person’s need for an intervention, use or Factor Surveillance, and the DHS’s routine collection of
exposure to an intervention, and if the intervention had its hemoglobin samples to test for anemia. In some countries,
intended effect (usually measured by a biological marker or health biological markers for sexually transmitted diseases (STDs) and
outcome) have to follow an information continuum, such that HIV also are collected [25].
these three factors can be linked together. Biological data collection can provide more objective measure-
Here we define the measure of intervention need as whether an ments of health needs, but it is important to consider diagnostic
individual would benefit from receiving a specific health interven- thresholds and limitations. There has been substantial debate over
tion. Intervention use reflects whether an individual, conditional the appropriate CD4 count thresholds for ART initiation [26],
on needing the intervention, received or used a specific and similar disagreements have occurred over threshold defini-
intervention. Intervention quality captures whether a specific tions for conditions such as hypercholesterolemia and hypergly-
intervention actually conferred the health gain or protection it was cemia [27]. This lack of consensus over threshold standards means
supposed to (effectiveness). To measure effective coverage of that estimating health needs can be highly influenced by whatever
diabetes management, for instance, information would need to be threshold is selected by clinicians and researchers.
collected on (1) the prevalence of diabetes in a population (i.e., In addition to surveys, surveillance data collected through
individuals who need treatment for diabetes); (2) the proportion of clinics and hospitals serve as another major source of health need
people with diabetes who receive treatment; and (3) the data. For instance, population-level estimates of HIV prevalence
effectiveness of their treatments (i.e., whether levels of fasting are often derived from routine blood tests conducted at antenatal
plasma glucose declined with treatment) [22]. clinics. These facility-based databases serve as convenient sources,
In the following sections, we will focus on discussing the but they may not provide fully representative information on
approaches and challenges for measuring each of these compo- broader population needs. Health facility records are unlikely to
nents. capture the health needs of individuals who do not regularly seek
health services (or the services provided by specific types of
Measuring Intervention Need facilities, such as antenatal clinics), so these sources of data tend to
under-represent the least wealthy populations [28].
Intervention need can be defined in different ways. First, Recent progress in verbal autopsy (VA) methods provides
intervention need can be viewed in normative terms. For instance, alternative tools for measuring health needs in resource-constrained
pregnant women would be considered the population in need of settings. For example, the Symptomatic Diagnosis (SD) approach
antenatal care, and children younger than one year old would be the generates a probabilistic ‘‘diagnosis’’ for conditions by using self-
population in need of the pentavalent vaccine. In places with well- reported symptoms collected during interviews. This approach can
developed health information systems, the number of pregnant supplement data collection when collecting biological markers or
women and young children in a population can be tracked with providing clinical assessments are not feasible. An SD pilot in
relative ease. However, in settings with less developed health Mexico has sought to identify cases of several non-communicable
information systems, this information may not be as easily attainable. diseases, including angina asthma, chronic obstructive pulmonary
Nationally representative surveys, such as the multi-country Demo- disease, vision loss, hearing loss, depression, and osteoarthritis [26].
graphic and Health Surveys (DHS), serve as efficient mechanisms for Preliminary results indicate that SD outperforms current question-
obtaining population information in places with less robust informa- naire-based epidemiological approaches in diagnosing diseases such
tion systems. In these surveys, retrospective information on pregnancy as depression, angina, and asthma [29]. The chance-corrected
and child births are routinely collected, which can offer insights into concordance was above 75% and as high as 93%. Further, the
recent needs and demand for relevant health services. However, these absolute error associated with SD applications was up to four times
surveys do not routinely capture all health needs that a population lower than current methods for some conditions.
might experience. The DHS, for example, primarily collects data on
communicable disease and maternal and child health conditions, Measuring Intervention Use Conditional on Need
largely excluding the measurement of non-communicable diseases,
mental health, and injury. As a result, routinely measuring health The use of an intervention is a central component of estimating
needs within these health domains is likely to require alternative effective coverage. Specifically, it is defined conditional on need.
strategies. In other words, it is essential to measure not only the number of
Second, intervention need can be determined by diagnosis, individuals using a service, but also differentiate the number of
which allows for the targeting of specific populations for individuals in need who are using it. When need is defined
interventions. For instance, individuals with HIV/AIDS need normatively, intervention use among those in need may be more

PLOS Medicine | www.plosmedicine.org 4 September 2014 | Volume 11 | Issue 9 | e1001730

 Table 1. Approaches to measuring effective coverage.

Potential data
Approach Description Study examples sources Strengths Limitations

Content - Focuses on the - WHO Quality - Hospital databases - Offers information from - Subjectivity in patient
of care health care process assessment and - Patient exit interviews both demand- and assessments of quality
- Involves indicators that assurance in primary supply-side factors - High outputs or content of care
target the resource and health care [37] - Resource and activity may not directly translate into
activity outputs of an outputs can serve as health gains
intervention objective indicators
Biomarkers - Focuses on the health - Assessment of vaccine - Health surveys that - Provides an objective - Collection of biomarker data
benefits that can be effectiveness [39] include physical measure of actual health can be costly and not always
detected biologically examinations gains or impact feasible in resource-constrained
settings
- Not applicable to all health
conditions
Cohort - Focuses on changes - Assessment of highly - Cohort registration - Provides measurement of - Limited to interventions that
registration in individual health active antiretroviral databases treatment effectiveness for involve close patient monitoring
outcomes over the therapy (HAART) [41] chronic conditions over time and treatment by healthcare
course of treatment providers
- Requires careful consideration
of time-dependent confounding
factors and lost to follow-up
Exposure - Compares health - Assessment of health - Household - Allows for the quantification - Household surveys are rarely
matching outcomes of individuals impact of IPTp survey data of the health gains associated powered to detect health effects
who had intervention and ITNs [43] with intervention exposure by - Unmeasured confounding
exposure to those who calculating odds ratios or factors need to be accounted for
did not have exposure relative risks with existing due to the observational nature
to an intervention data of analysis
Statistical - Uses statistical and - Assessment of - Health survey data - Offers a convenient solution - Only approximates the
methods econometric techniques, diabetes and hypertension to address potential biases relationship, or correlation,
such as instrumental management in Iran [45] associated with confounding between intervention exposure
variables (IVs) and factors and a health outcome rather
matching, to estimate than the causal effect
health outcomes while
controlling for
unobserved variables
Risk-adjusted - Estimates health - Birth weight–adjusted - Hospital databases - Provides an indicator for - Limited to interventions that
outcomes outcomes while neonatal mortality [46] quality of care that reflects are delivered at health facilities
accounting both procedural outputs - Certain risks may not be easily
for the patient and the health impact of adjusted for if they are
characteristics and risks received care challenging to quantify
of death that can vary
systematically across sites

doi:10.1371/journal.pmed.1001730.t001

easily measured by the total number of individuals belonging to a triangulated data or combined estimates of intervention use from
particular demographic category who have accessed an interven- multiple sources through statistical modeling [31].
tion, for example the number of pregnant women who have Different strategies for data validation and synthesis are
attended antenatal care (ANC) or the number of children younger regularly used to estimate trends in intervention coverage. For
than 12 months old who have received the pentavalent vaccine. example, expert groups have assessed the most appropriate
Properly measuring intervention use or exposure and tracking analytical methods for WHO and UNICEF to use in estimating
intervention coverage over time can be challenging, especially immunization intervention coverage [32]. In other cases, system-
because intervention data sources are often subject to inconsis- atically testing different modeling strategies has been the
tencies and information gaps [30]. predominant approach. For example, a Bayesian model applied
Data on intervention use can be extracted from several sources, a systems dynamic framework to bring together multiple sources of
including administrative systems and household surveys. Admin- data on ITNs, ranging from ITN delivery records from
istrative health databases generally offer the most complete records manufacturers to household survey measures of ITN ownership,
of intervention use over time (e.g., the number of insecticide- to construct annual estimates of ITN coverage [33–34]. This
treated nets [ITNs] distributed each year), which is immensely modeling approach demonstrated how capturing multiple mea-
helpful for computing trends in intervention coverage. At the same sures along a distribution chain can support the annual estimation
time, administrative sources often experience a variety of reporting of intervention coverage.
biases and may not link the receipt of an intervention to an
individual’s need for it. Household surveys generally provide more Measuring Intervention Quality
robust estimates of intervention coverage, but the gaps in time
between survey administrations can make tracking intervention Capturing whether the intended health benefit was provided by
trends difficult. As a result, many studies and programs have an intervention is what differentiates effective coverage from more

PLOS Medicine | www.plosmedicine.org 5 September 2014 | Volume 11 | Issue 9 | e1001730

traditional ‘‘crude’’ intervention coverage. This requirement for controlling for time-dependent covariates including CD4 counts
additional information, which is generally measured by a and HIV-1 RNA concentration, it was found that HAART
biological marker or observable health outcome, makes effective substantially reduced the rate of progression to AIDS or deaths by
coverage more challenging to assess than coverage alone. 86%. Another issue that requires proper attention when analyzing
Although assessing intervention quality is often the most compli- cohort data is when patients are lost to follow-up. A recent study
cated aspect of estimating effective coverage, several approaches examining the effect of lost to follow-up among HIV-infected
have been proposed and tested under routine settings [35–36]. individuals showed that estimated survival with and without
These approaches include content of care, biomarkers, cohort adjustment of mortality amongst those lost to follow-up can lead to
registration, exposure matching, statistical methods, and risk- overestimation of the treatment effect by as much as 40% [41].
adjusted outcomes. Table 1 gives an overview of the strengths and Despite the rich information contained in cohort registration,
limitations of the approaches, which all provide a quantitative maintenance of the system can be costly.
means to estimating the health gain associated with the receipt or
use of a specific intervention. Exposure Matching
Effectiveness of an intervention can also be inferred using
Content of Care household survey data by comparing the health outcomes of
Content of care focuses on the health care process. It often individuals with and without exposure to the intervention. This
involves indicators that target the resource and activity outputs of strategy is known as exposure matching. Exposure matching has
an intervention [37]. Information on these indicators can be been applied in a variety of contexts, including the assessment of
obtained through both providers and beneficiaries. Asking the effectiveness of malaria control interventions. Using household
respondents about the frequency, timing, and content of antenatal survey data, a recent study compared birth weight and survival of
care in household surveys, including the type of provider and type infants whose mothers had had different exposures to ITN and/or
of tests accompanying ANC visits, is an example of a beneficiary- intermittent preventive therapy during pregnancy (IPTp). The
based approach. However, one of the major limitations of using study found that exposure to full malaria prevention with IPTp or
content of care as a measure of quality is that high content of care ITNs reduced the risk of neonatal mortality and the odds of low
does not always associate with positive health gain. This birth weight by 18% and 21%, respectively [42].
discrepancy is demonstrated in a recent study on patient It is important to note, however, that household surveys are
satisfaction and health outcomes conducted in the United States often not powered to detect health effects in this way. As a result,
[38]. Using national survey data, the study found that high patient although aggregate level estimates of intervention effects across
satisfaction was not associated with positive health outcome but countries are detectable, variation across countries is less easily
rather higher mortality. measured. To facilitate the measurement of intervention effec-
tiveness at a finer scale, increasing in sample size for household
Biomarkers surveys on a periodic basis can be considered. Another important
One way to objectively measure quality in terms of the actual caveat when carrying out exposure matching using survey data is
health benefit of an intervention is to evaluate biomarkers. This the presence of unmeasured confounding factors as the analyses
approach was used in a study comparing the effectiveness of a one- are observational retrospective in nature.
versus two-dose regimen of measles-mumps-rubella (MMR)
vaccine [36]. Single-dose MMR vaccine is common in many Statistical Methods
countries. Although clinical studies have demonstrated poor As illustrated in the previous sections, because of the presence of
efficacy for single-dose MMR, the effectiveness (or the lack potential confounding factors, it is difficult to immediately
thereof) of the regimen at the population level has not been attribute a health outcome to the effectiveness of an intervention
investigated empirically. By evaluating the biomarker levels, in retrospective or cross-sectional analyses. One way to tackle this
specifically MMR IgM and IgG antibody, of over 1,000 children limitation is to apply statistical and econometric methodologies,
who had received one- versus two-dose of MMR, researchers such as instrumental variable (IV) and matching. IV was used in a
showed that measles and mumps IgG antibody levels were recent study assessing the effectiveness of influenza vaccination
considerably lower than putative levels among children who had among elderlies in Ontario, Canada [43]. Contrary to previous
only one dose of the vaccine. This study provided empirical research, the study found no significant association between
evidence in support of reinforcing a two-dose vaccine regimen in influenza vaccination and all-cause mortality. In another study,
order to achieve population-level immunity [39]. However, the effectiveness of diabetes and hypertension management
obtaining biomarker measures is not always feasible due to intervention in Iran using mixed-effect models and propensity
resource constraints as well as the nature of diseases. score matching [22]. The results suggested an association between
the intensity of primary diabetes and hypertension management
Cohort Registration intervention and improved health outcomes. Although these
For interventions that involve close patient monitoring and statistical techniques serve as convenient solutions to the issue of
treatment by care providers, the most fitting approach for tracking confounding factors, they only approximate the correlation, rather
effectiveness is cohort registration. One example is the WHO’s than causation, between intervention and effect.
strategy for treating tuberculosis, known as DOTS (directly
observed treatment, short-course). Through the program, new Risk-Adjusted Outcomes
cases of tuberculosis are recorded; adherence and treatment For intervention delivered in hospital settings, one approach to
outcomes are also documented. By evaluating individuals’ capture effectiveness is to estimate the risk-adjusted outcomes.
information over time, one can obtain direct quality measure of Risk-adjusted outcomes such as risk-adjusted hospital mortality
the effectiveness of intervention. When analyzing cohort data, and risk-adjusted 29-day mortality serve as proxies for the quality
time-dependent confounding factors must be taken into consider- of hospital care while taking into account the fact that the patient
ation. A good example of this approach is the study of highly characteristics and risks of death may vary systematically across
active antiretroviral therapy (HAART) by Sterne et al [40]. By sites. In a recent study by Straney et al [44], the variation in

PLOS Medicine | www.plosmedicine.org 6 September 2014 | Volume 11 | Issue 9 | e1001730

 Box 2. Recommendations application. In many settings, one of the major obstacles is a lack
of reliable data. Strengthening health information system capacity
1. Identify disease burden, affordable interventions, should occur alongside the implementation of health reforms for
and social priorities when selecting which inter- achieving UHC. Ideally, health information systems would
ventions to include in estimating effective cover- capture both supply- and demand-side data on health services,
age and then support the triangulation of data. The need for further
strengthening of health information systems is not limited to lower-
N Examine existing evidence on health need priorities. resource settings; it is a challenge experienced by higher- and
N Evaluate cost-effectiveness of interventions. lower-income countries alike [45–47].For instance, a United
States–based study recently created a population-based health
N Consider concerns of equity: political, social, and
surveillance system for tracking disparities in chronic diseases
cultural
across ethnic groups in King County, Washington [48]. This
2. Develop measurement strategies for tracking system sought to integrate multiple sources of administrative data,
need, use, and quality for selected interventions including medical discharge records, reportable conditions, payer
data, and Medicare files, with data collected from surveys and
N Intervention need can be measured using existing physical examinations. Through these triangulation efforts, King
survey data, biological markers, or alternative methods County had a more accurate mechanism for tracking the health
such as Symptomatic Diagnosis (SD). needs of its entire population, relying less on individual contact
N Intervention use can be estimated by synthesizing with the health system. A similar integrative health information
administrative and household survey data. system is under development in the Kingdom of Saudi Arabia
N Intervention quality can be determined by different [49].
methods that link health outcomes with the receipt or Because health policies are often being driven and executed at a
use of interventions. Examples of these approaches local-level, the ideal integrated health information system should not
include exposure matching, risk-adjusted health out- only be designed to enable estimations of effective coverage at the
comes, and statistical modeling. national level, it should also allow estimations of the metric at the
subnational level [16,50–53]. Derivations of effective coverage at
3. Build capacity for measuring effective coverage subnational level have been challenging because of the lack of
dependable and representative data sources. Limitations in human
N Develop an integrated health surveillance system that
resources and technical supports hinder regular collection of health
allows triangulation of data.
data at the local level. Moreover, many surveys implemented
N Devote resources for training staff on data collection
sampling schemes that are designed to be representative at the
and basic analysis.
national level only. In some cases, post-stratification weighing
strategy can be applied. However, that does not guarantee that the
estimates accurately represent the actual local demographic
neonatal clinical care across the US was investigated. By composition. Advancement in small area methodologies, which
examining the gestational age and birth weight-adjusted (GA- capitalized on geographical relatedness, has substantially enhanced
BW) neonatal mortality from 1960–2006, the study concluded that our capacity to maximize the use of existing data to estimate and
the quality of obstetrical and neonatal care have indisputably evaluate disease prevalence and intervention coverage at the local
improved over time. One of the limitations of risk-adjusted level [54]. Nevertheless, a solid data collection platform is crucial for
outcomes, however, is that certain risks may not be easily long-term monitoring and policy planning.
quantifiable and hence adjusted for.
It is important to emphasize that the notion of quality in Conclusions
effective coverage not only reflects the content pertaining to health
services, which measures the resource and activity outputs of an UHC entails a global health ideal which, with concerted effort,
intervention [37]. More importantly, it directly captures interven- could become reality. At the same time that policymakers and
tion impact on health under routine conditions. Capturing health health officials are developing strategic plans for achieving UHC
impact in the quality component is critical because high levels of for their countries, it is also critical to prioritize establishing a data-
content do not necessarily translate into optimal health outcomes driven framework for tracking progress in achieving UHC. In
and impact. Effective coverage measured in this manner can help doing this, governments foster responsiveness and accountability
more precisely pinpoint gaps in health service delivery, especially if for their UHC aims.
levels of effective coverage are compared with crude measures of In this paper, we have reviewed the concept of effective
intervention coverage. For example, if two districts have similar coverage and highlighted three main components that affect its use
crude coverage of 80% for an intervention, but one has 75% under routine conditions. Box 2 provides recommendations on the
effective coverage and the other has 50% effective coverage of the major considerations for tracking effective coverage. These include
same intervention, this finding would imply differential delivery first, reviewing existing evidence on disease burden, affordable
and quality of care. Thus, the policy recommendation for the two interventions and social priorities; second, developing strategies to
districts will vary dramatically. Effective coverage is designed to be measure needs, use, and quality; and third, building system
a flexible and powerful health metric that can uniquely help to capacity for continuous monitoring. Among these considerations,
understand actual health system performance. building capacity for data collection and use remains the most
substantial hurdle in broadly using effective coverage. Without
Building Capacity for Tracking Effective Coverage further developing the strength and representation of routine
health information systems, tracking national and subnational
Having both the infrastructure and human resources needed to progress towards health goals, such as UHC, is likely to be more
optimally track effective coverage is critical to the metric’s resource-intensive and prone to suboptimal accuracy.

PLOS Medicine | www.plosmedicine.org 7 September 2014 | Volume 11 | Issue 9 | e1001730

 As health reform efforts continue to evolve worldwide, the range Author Contributions
and scope of interventions comprising UHC priorities are likely to
Wrote the first draft of the manuscript: MN. Contributed to the writing of
change over time as well. By harnessing existing health informa- the manuscript: MN ADF NF CJLM SSL JLD. ICMJE criteria for
tion systems and expanding their capacity, countries can be in the authorship read and met: MN ADF NF CJLM SSL JLD. Agree with
position of using effective coverage to align with their own UHC manuscript results and conclusions: MN ADF NF CJLM SSL JLD.
needs and to more accurately monitor progress towards their
UHC goals.

References
1. World Health Organization (2010) The world health report - Health systems 24. Boerma JT, Holt E, Black R (2001) Measurement of Biomarkers in Surveys in
financing: the path to universal coverage. WHO. Available: http://www.who. Developing Countries: Opportunities and Problems. Popul Dev Rev 27: 303–
int/whr/2010/en/. Accessed 24 October 2013. 314. doi:10.1111/j.1728-4457.2001.00303.x
2. Bennett S, Ozawa S, Rao KD (2010) Which Path to Universal Health 25. Macro International Inc. (2009) MEASURE DHS. Demographic and Health
Coverage? Perspectives on the World Health Report 2010. PLoS Med 7: Surveys. Available: http://www.measuredhs.com/.
e1001001. doi:10.1371/journal.pmed.1001001 26. World Health Organization (2013) Consolidated ARV guidelines 2013 WHO.
3. World Health Organization (2013) World health report 2013: Research for Available: http://www.who.int/hiv/pub/guidelines/arv2013/art/
universal health coverage. WHO. Available: http://www.who.int/whr/en/. statartadolescents_rationale/en/. Accessed 11 March 2014.
Accessed 24 October 2013. 27. Kaplan RM, Ganiats TG, Frosch DL (2004) Diagnostic and Treatment
4. Alfonso EA, Diaz Y, Giedion U (2013) The impact of universal coverage Decisions in US Healthcare. J Health Psychol 9: 29–40. doi:10.1177/
schemes in the developing world: a review of the existing evidence. The World 1359105304036100
Bank. Available: http://documents.worldbank.org/curated/en/2013/01/ 28. Makinen M, Waters H, Rauch M, Almagambetova N, Bitran R, et al. (2000)
17291221/impact-universal-coverage-schemes-developing-world-review- Inequalities in health care use and expenditures: empirical data from eight
existing-evidence. Accessed 22 October 2013. developing countries and countries in transition. Bull World Health Organ 78:
5. World Health Organization (n.d.) Monitoring universal health coverage WHO. 55–65. doi:10.1590/S0042-96862000000100006
Available: http://www.who.int/healthinfo/country_monitoring_evaluation/ 29. James SL, Lozano R, Romero M, Gómez S, Ramı́rez D, et al. (2013) Can
universal_health_coverage/en/. Accessed 8 March 2014. computers measure the chronic disease burden using survey questionnaires? The
6. Campbell J, Buchan J, Cometto G, David B, Dussault G, et al. (2013) Human Symptomatic Diagnosis Study. Lancet 381: S65. doi:10.1016/S0140-
resources for health and universal health coverage: fostering equity and effective 6736(13)61319-7
coverage. Bull World Health Organ 91: 853–863. doi:10.2471/BLT.13.118729 30. Lim SS, Stein DB, Charrow A, Murray CJ (2008) Tracking progress towards
7. Knaul FM, González-Pier E, Gómez-Dantés O, Garcı́a-Junco D, Arreola- universal childhood immunisation and the impact of global initiatives: a
Ornelas H, et al. (2012) The quest for universal health coverage: achieving social systematic analysis of three-dose diphtheria, tetanus, and pertussis immunisation
protection for all in Mexico. Lancet 380: 1259–1279. doi:10.1016/S0140- coverage. Lancet 372: 2031–2046. doi:16/S0140-6736(08)61869-3
6736(12)61068-X 31. Roberts TJ, Carnahan E, Gakidou E (2013) Can breastfeeding promote child
8. O’Connell T, Rasanathan K, Chopra M (2014) What does universal health health equity? A comprehensive analysis of breastfeeding patterns across the
coverage mean? Lancet 383: 277–279. doi:10.1016/S0140-6736(13)60955-1 developing world and what we can learn from them. BMC Med 11: 254.
9. Murray CJL (2003) Health Systems Performance Assessment: Debates, Methods doi:10.1186/1741-7015-11-254
and Empiricism. World Health Organization. 919 p. 32. Burton A (2009) WHO and UNICEF estimates of national infant immunization
10. Lozano R, Soliz P, Gakidou E, Abbott-Klafter J, Feehan DM, et al. (2006) coverage: methods and processes. Bull World Health Organ 87: 535–541.
Benchmarking of performance of Mexican states with effective coverage. Lancet doi:10.2471/BLT.08.053819
368: 1729–1741. doi:10.1016/S0140-6736(06)69566-4
33. Flaxman AD, Fullman N, Otten MW Jr, Menon M, Cibulskis RE, et al. (2010)
11. Lozano R, Naghavi M, Foreman K, Lim S, Shibuya K, et al. (2012) Global and
Rapid scaling up of insecticide-treated bed net coverage in Africa and its
regional mortality from 235 causes of death for 20 age groups in 1990 and 2010:
relationship with development assistance for health: a systematic synthesis of
a systematic analysis for the Global Burden of Disease Study 2010. Lancet 380:
supply, distribution, and household survey data. PLoS Med 7: e1000328.
2095–2128. doi:10.1016/S0140-6736(12)61728-0
doi:10.1371/journal.pmed.1000328
12. Sachs JD (2012) Achieving universal health coverage in low-income settings.
34. World Health Organization (2012) World Malaria Report 2011 WHO.
Lancet 380: 944–947. doi:10.1016/S0140-6736(12)61149-0
Available: http://www.who.int/malaria/publications/atoz/9789241564403/
13. Ozgediz D, Hsia R, Weiser T, Gosselin R, Spiegel D, et al. (2009) Population Health
en/index.html. Accessed 1 November 2013.
Metrics for Surgery: Effective Coverage of Surgical Services in Low-Income and
35. Donabedian A (2005) Evaluating the Quality of Medical Care. Milbank Q 83:
Middle-Income Countries. World J Surg 33: 1–5. doi:10.1007/s00268-008-9799-y.
691–729. doi:10.1111/j.1468-0009.2005.00397.x
14. Sullivan R, Peppercorn J, Sikora K, Zalcberg J, Meropol NJ, et al. (2011)
Delivering affordable cancer care in high-income countries. Lancet Oncol 12: 36. Donabedian A (1978) The quality of medical care. Science 200: 856–864.
933–980. doi:10.1016/S1470-2045(11)70141-3 37. Roemer MI, Montoya-Aguilar C, Organization WH (1988) Quality assessment and
15. Murray CJL, Richards MA, Newton JN, Fenton KA, Anderson HR, et al. (2013) assurance in primary health care/M. I. Roemer and C. Montoya-Aguilar. Available:
UK health performance: findings of the Global Burden of Disease Study 2010. http://apps.who.int/iris/handle/10665/40663. Accessed 2 November 2013.
Lancet 381: 997–1020. doi:10.1016/S0140-6736(13)60355-4 38. Fenton JJ, Jerant AF, Bertakis KD, Franks P (2012) The cost of satisfaction: A national
16. Murray CJL, Abraham J, Ali MK, Alvarado M, Atkinson C, Baddour LM, et al study of patient satisfaction, health care utilization, expenditures, and mortality. Arch
(2013) The state of us health, 1990-2010: Burden of diseases, injuries, and risk Intern Med 172: 405–411. doi:10.1001/archinternmed.2011.1662
factors. JAMA 310: 591–608. doi:10.1001/jama.2013.13805 39. Pebody R, Gay N, Hesketh L, Vyse A, Morgan-Capner P, et al. (2002)
17. Yang G, Wang Y, Zeng Y, Gao GF, Liang X, et al. (8) Rapid health transition in Immunogenicity of second dose measles–mumps–rubella (MMR) vaccine and
China, 1990–2010: findings from the Global Burden of Disease Study 2010. implications for serosurveillance. Vaccine 20: 1134–1140. doi:10.1016/S0264-
Lancet 381: 1987–2015. doi:10.1016/S0140-6736(13)61097-1. 410X(01)00435-2
18. World Health Organization (n.d.) Cost effectiveness and strategic planning 40. Sterne JA, Hernán MA, Ledergerber B, Tilling K, Weber R, et al. (2005) Long-
(WHO-CHOICE). WHO. Available: http://www.who.int/choice/en/. Ac- term effectiveness of potent antiretroviral therapy in preventing AIDS and death:
cessed 18 December 2013. a prospective cohort study. Lancet 366: 378–384. doi:10.1016/S0140-
19. Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, et al, editors 6736(05)67022-5
(2006) Disease Control Priorities in Developing Countries. 2nd ed. Washington 41. Verguet S, Lim SS, Murray CJL, Gakidou E, Salomon JA (2013) Incorporating
(DC): World Bank. Available: http://www.ncbi.nlm.nih.gov/books/ Loss to Follow-up in Estimates of Survival Among HIV-Infected Individuals in
NBK11728/. Accessed 18 December 2013. Sub-Saharan Africa Enrolled in Antiretroviral Therapy Programs. J Infect Dis
20. Saltman RB, Ferroussier-Davis O (2000) The concept of stewardship in health 207: 72–79. doi:10.1093/infdis/jis635
policy. Bull World Health Organ 78: 732–739. 42. Eisele TP, Larsen DA, Anglewicz PA, Keating J, Yukich J, et al. (2012) Malaria
21. Victora CG, Hanson K, Bryce J, Vaughan JP (2004) Achieving universal prevention in pregnancy, birthweight, and neonatal mortality: a meta-analysis of
coverage with health interventions. Lancet 364: 1541–1548. doi:10.1016/ 32 national cross-sectional datasets in Africa. Lancet Infect Dis 12: 942–949.
S0140-6736(04)17279-6 doi:10.1016/S1473-3099(12)70222-0
22. Farzadfar F, Murray CJ, Gakidou E, Bossert T, Namdaritabar H, et al. (7) 43. Wong K, Campitelli MA, Stukel TA, Kwong JC (2012) Estimating influenza
Effectiveness of diabetes and hypertension management by rural primary health- vaccine effectiveness in community-dwelling elderly patients using the
care workers (Behvarz workers) in Iran: a nationally representative observational instrumental variable analysis method. Arch Intern Med 172: 484–491.
study. Lancet 379: 47–54. doi:10.1016/S0140-6736(11)61349-4 doi:10.1001/archinternmed.2011.2038
23. Fenton KA, Johnson AM, McManus S, Erens B (2001) Measuring sexual 44. Straney LD, Lim SS, Murray CJL (2012) Disentangling the Effects of Risk
behaviour: methodological challenges in survey research. Sex Transm Infect 77: Factors and Clinical Care on Subnational Variation in Early Neonatal Mortality
84–92. doi:10.1136/sti.77.2.84 in the United States. PLoS ONE 7: e49399. doi:10.1371/journal.pone.0049399

PLOS Medicine | www.plosmedicine.org 8 September 2014 | Volume 11 | Issue 9 | e1001730

 45. Yip WC-M, Hsiao WC, Chen W, Hu S, Ma J, et al. (2012) Early appraisal of 50. Hajjar I, Kotchen TA (2003) Trends in prevalence, awareness, treatment, and
China’s huge and complex health-care reforms. Lancet 379: 833–842. control of hypertension in the United States, 1988-2000. JAMA 290: 199–206.
doi:10.1016/S0140-6736(11)61880-1 doi:10.1001/jama.290.2.199
46. Reddy KS, Patel V, Jha P, Paul VK, Kumar AS, et al. (2011) Towards 51. Fang P, Dong S, Xiao J, Liu C, Feng X, et al. (2010) Regional inequality in
achievement of universal health care in India by 2020: a call to action. Lancet health and its determinants: Evidence from China. Health Policy 94: 14–25.
377: 760–768. doi:10.1016/S0140-6736(10)61960-5 doi:10.1016/j.healthpol.2009.08.002
47. Frenk J, González-Pier E, Gómez-Dantés O, Lezana MA, Knaul FM (2006) 52. Barraza-Lloréns M, Bertozzi S, González-Pier E, Gutiérrez JP (2002) Addressing
Comprehensive reform to improve health system performance in Mexico. Inequity In Health And Health Care In Mexico. Health Aff (Millwood) 21: 47–
Lancet 368: 1524–1534. doi:10.1016/S0140-6736(06)69564-0 56. doi:10.1377/hlthaff.21.3.47
48. Monitoring Disparities in Chronic Conditions Study: The MDCC Study (2013). 53. Mackenbach JP, Kunst AE, Cavelaars AE, Groenhof F, Geurts JJ (1997)
Available: http://www.healthmetricsandevaluation.org/research/project/ Socioeconomic inequalities in morbidity and mortality in western Europe.
monitoring-disparities-chronic-conditions-study-mdcc-study. Accessed 30 Octo- Lancet 349: 1655–1659. doi:10.1016/S0140-6736(96)07226-1
ber 2013. 54. Srebotnjak T, Mokdad AH, Murray CJ (2010) A novel framework for validating
49. Kingdom of Saudi Arabia Health Tracking (2013). Available: http://www.
and applying standardized small area measurement strategies. Popul Health
healthmetricsandevaluation.org/research/project/kingdom-saudi-arabia-
Metr 8: 26. doi:10.1186/1478-7954-8-26
health-tracking. Accessed 30 October 2013.

PLOS Medicine | www.plosmedicine.org 9 September 2014 | Volume 11 | Issue 9 | e1001730

View publication stats