An Introduction to the Grade of Confidence Used to Characterize Uncertainty Around the WHO and UNICEF Estimates of National Immunization Coverage

Uncertainty is a complex phenomenon and is nearly ubiquitous, at least in real-world settings [2]. Depending on the domain in which one works, the definition of ‘uncertainty’ may differ slightly. We consider uncertainty to be a state of limited or imperfect knowledge about a measurement, event or outcome. This uncertainty may derive from incomplete data, contextual factors that indirectly influence or perhaps conflict with existing information as well as from other known and perhaps unknown sources [3]. It is important to note, however, that uncertainty does not necessarily imply that an inference or judgement is incorrect [4].

The characterization, representation and communication of uncertainty are major issues in many domains, and there are multiple approaches to dealing with or reasoning under uncertainty. Some of the approaches include use of analogy [2], mental simulation, non-statistical characterization of degrees of credibility (e.g., use of bounds [possible/impossible], rough sets [possible/doubtful/impossible], fuzzy sets) [4] as well as statistical procedures such as Monte Carlo simulations and computational modelling. A review of (known) approaches, their advantages and disadvantages, is beyond the scope of this note, and we refer the reader to existing reviews [6,7] of the many approaches to the treatment of uncertainty.
*Address correspondence to this author at the UNICEF House, Three UN Plaza, New York, New York, USA 10017; Tel: +1.212.303.7010; E-mail: dbrown@unicef.org Uncertainty belies the description of immunization system performance as measured through immunization coverage. Our purpose in this note is to introduce a Grade of Confidence (GoC) used to characterize uncertainty around the WHO and UNICEF estimates of national immunization coverage. Since 2000 the World Health Organization (WHO) and United Nations Children's Fund (UNICEF) have annually estimated national infant immunization coverage for WHO recommended vaccines (see www.who.int/immunization/policy/Immunization_routine_table2.pdf) for 195 countries or territories. The methodology of the estimates and a description of the data used to inform the estimates is described elsewhere [8,9].
The WHO and UNICEF estimates of national immunization coverage (wuenic) are based on data and informationincluding reports by national immunization programmes regarding immunization coverage for select antigens as well as survey data from the published and grey literature [8] that are of varying, and, in some instances, unknown quality. In order to improve the contribution of the wuenic to decision making, we feel it is important to communicate the uncertainties associated with the wuenic.
Beginning with the 2011 revision (completed July 2012) of the WHO and UNICEF estimates, a GoC was introduced as a means of conveying the uncertainty in these estimates. Importantly, the use of the term 'confidence' here does not imply any reference to measurement error or statistical uncertainty in the underlying empirical data. As there is no underlying probability model upon which the wuenic are based, we are unable to present classical measures of statistical uncertainty, e.g., confidence intervals. Moreover, we have chosen not to make subjective estimates [10] of plausibility/certainty ranges around the wuenic, which may be insensitive to some factors and sensitive to others [11] and pose deeper issues in quantitative estimation [12].
The approach taken can be thought of as using a model of "endorsement" similar to that used in the artificial intelligence literature [1]. Endorsements are defined as "…reasons to believe or disbelieve propositions and are the basis of explanations and control decisions in uncertain reasoning" [1]; alternatively, endorsements might be considered as records of information that may influence one's certainty about an inference [1,5]. The "confidence", or certainty, in the inferences or decisions that are made is proportional to the accumulation of endorsements [5].
Currently, three endorsements are possible in the GoC: 1) the WHO and UNICEF coverage estimate is based on data reported by national authorities [8], 2) the WHO and UNI-CEF coverage estimate is supported by (i.e., < 10%-points away from) coverage levels recomputed using the number of children vaccinated as reported by national authorities and the number of births or surviving infants from the UN Population Division 1 ; and 3) the WHO and UNICEF coverage estimate is supported by survey results within two years (before or after). Using this approach, the degrees of empirical support upon which the wuenic are based are classified as follows: *** The WHO and UNICEF coverage estimate is supported by (i.e., < 10%-point difference) For example, in Fig. (1) example (a), the WHO and UNICEF coverage estimate is supported by (i.e., < 10%points) recomputed coverage using numerator data from the government and independent denominator and the WHO and UNICEF estimate is not challenged by any other data source during 2003,2004,2005. As such the wueinc GoC takes a value of "two stars" (**). However, in the other years shown in the figure, the WHO and UNICEF coverage estimate is challenged by (i.e., differs by >10% points from) the recomputed coverage levels and therefore the wuenic GoC takes a value of "one star" (*).
In the Example (b) of Fig. (1), during 2005-2009 the WHO and UNICEF coverage estimate is supported by (i.e., < 10%-points) data reported by national authorities, recomputed coverage, and at least one supporting survey (a survey for the 2007 birth cohort) within two years of the estimate and there is no survey that challenges the estimate. The wuenic GoC for 2005-2009 in this case take a value of "three stars" (***) representing high confidence in the wuenic value. During 2004 and 2010-2012, the WHO and UNICEF coverage estimate is supported by recomputed coverage and the WHO and UNICEF estimate is not challenged by any other data source; therefore, wuenic GoC takes a value of "two stars" (**). During 2003, the WHO and UNI-CEF estimate is supported by data reported by national authorities and is not challenged by any other data source leading to a wuenic GoC value of "two stars" (**). However, in 2001-2002, the WHO and UNICEF coverage estimate is challenged by (i.e., differs by >10% points from) the recomputed coverage levels and therefore the wuenic GoC takes a value of "one star" (*) representing low confidence in the wuenic value. It is useful to note that the survey shown for the 2001 birth cohort was ignored due to a small sample size (< 300 observations); however, even if this survey had not been ignored, the recomputed coverage level would have challenged the WHO and UNICEF estimate.
The wuenic GoC is not a judgment of the quality of data reported by national authorities that serve as one of several inputs to the wuenic. Even a wuenic with a high GoC (***) carries a risk of being wrong despite being well supported. In all cases, the wuenic should be used with caution and should be assessed in light of the objective for which they are being used.
The wuenic GoC continues to be refined and is subject to limitations. The taxonomy used for the GoC is only one of many possible alternatives. The GoC is currently based on a subset of available evidence, which may differ from estimate to estimate. Evidence such as expert judgement from field observations or disease surveillance is currently not included. To the extent that more evidence is available to inform an estimate, there are also increased opportunities for conflicting evidence (i.e., challenges) to come into play. In addition, we currently do not discriminate among, or rank, different kinds of evidence. Survey evidence, on the one hand, and coverage recomputed using a national reported numerator and independent denominator, on the other hand, are given equal consideration. Within available survey evidence, survey coverage based on 350 observations is treated similarly to that from a survey of 5,000 observations. In the same manner, a survey with 30% vaccination evidence documented by cards is considered equal to one with 80% documented vaccination evidence. Also, WHO and UNICEF estimates for the most recent two periods in time are often less likely to have evidence from alternative data sources, in particular surveys.
In summary, growing interest in estimates of national immunization coverage is accompanied by an increasing demand for a "precise representation of what is [often] in effect a vague magnitude" [12]. As such, consideration of uncertainties around the WHO and UNICEF estimates of national immunization coverage is important. The grade of confidence (GoC) is an attempt to characterize uncertainty in wuenic as a function of the endorsements or empirical support that an estimate has received. Currently three endorsements are possible. Future refinements to the wuenic GoC may include incorporation of additional endorsements and consideration of differentiation of empirical support.

CONFLICT OF INTEREST
The author(s) confirm that this article content has no conflicts of interest.