Applying Formal Consensus Methods To Enhance The Credibility Of Public Health Guideline Development – A Case Study

2.1. Consensus Methods Literature Review: Search Strategy

The original search was undertaken during June 2024 and repeated in December 2024 (S3). The project team identified a total of 791 potentially relevant records from a systematic search of Medline, Embase, and CINAHL, of which 409 duplicates were removed at this point. A total of 382 records were then screened in Rayyan [27] by title and abstract, with a further 241 excluded on the basis of incorrect outcome, population, study design, foreign language, or duplication. The remaining 141 records were subjected to full-text screening by two authors using Rayyan in blinded mode. A thematic analysis of full-text records was applied based on additional inclusion criteria that considered consensus articles reporting on the risk of bias in group decision-making, ensuring transparency in the process, systematic methods adopted, and a comprehensive perspective in decision-making. [28] Of these, a further 70 records were excluded, leaving 71 remaining. A hand search also yielded a further 15 records, therefore, a final total of 86 records were included in the review. [See Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram (PRISMA) Fig. (1).

The search strategy was reviewed by the lead author and the Knowledge Management Officer (RGDU). Any differences of opinion between reviewing authors were resolved by consensus discussion.

2.2. Defining Consensus

It has been previously espoused that there is no requirement to characterise consensus as unanimity among all guideline development group (GDG) members [17]. To that end, many studies have purposely formulated, a priori, definitive rules that aggregate results, thus indicating the strength of agreement. For this case study, the research team adopted the meaning of consensus purported by the World Health Organisation (2014) [4] “In guideline development groups, consensus decision-making is a process whereby the consent of all committee members is pursued. When consensus has been reached, it generally means that every committee member finds the proposed resolution acceptable – or at least lends it support, even if less than wholeheartedly”.

This understanding of consensus is further underlined by Siwiec and colleagues (2019) [18], who emphasize the role of negotiation as an instrument for achieving formal consensus. Rather than pursuing concessions from panel members, it is argued that the purpose of consensus is recognising solutions that are acceptable to all involved. [18] Similarly, Nair et al. (2011) [17] have affirmed that perfect consensus is seldom realized; therefore, the objective of consensus methods is to establish a central tendency within the group and grade the resultant level of agreement.

2.3. Membership of Guideline Development Panels

Participants in this project included those who were members of the GDG and subsequently formed subject matter expert topic groups (SME-TGs). Membership was established on the basis of several key principles, supported by the World Health Organization, to ensure that the panels provide credible, balanced, and relevant public health guidance. [29] These included relevant subject matter, clinical expertise and experience, relevant academic expertise, diversity of knowledge and experience, commitment and participation, and conflict of interest disclosure. Although participation bias is to some extent inevitable in populating public health expert panels, this was mitigated by recruiting panel members from a broad range of backgrounds, specialties, and experiences to ensure no single group dominated the discussions. [30]

The Terms of Reference were ratified for the GDG, delineating the role of the Chair, the responsibilities of group members, and the responsibilities of the RGDU. This document also outlined principles for working together, which included openness, respect, transparency, and confidentiality (S4). The GDG was chaired by a subject matter expert/Consultant in Public Health Medicine, with methodological expertise provided by the RGDU. The GDG comprised twenty-seven multidisciplinary members. The records from GDG meetings reflect a consistent picture among group members throughout the project's lifetime. This was important for ensuring continuity, cohesion, and high-quality decision-making, particularly with reference to consensus panels. [31] The rationale for the number of panel members was to ensure an optimal group size to achieve a topic-appropriate balance of expertise and adequate representation on the guideline panel and SME-TGs. An iterative process was initiated with membership reviewed at all meetings to ensure requisite experience and content expertise throughout the guideline development process.

Membership of the GDG reflected stakeholders from across the entire spectrum of national public health, including infectious diseases (adult and paediatric), microbiology, Irish College of General Practitioners (ICGP), Antimicrobial Resistance and Infection Control (AMRIC) ¹, National Social Inclusion Office (NSIO), Irish Prison Services (IPS), National Virus Reference Laboratory (NVRL), National Immunisation Office (NIO), Pharmacy and Occupational Health.

Members were recruited to the GDG via email by the National Measles Incident Management Team (N-IMT), a collaborative national structure convened by the Director of National Health Protection in response to the measles threat in early 2024. Members had experience, knowledge, and skills in evidence-based medicine and guideline development. The national RGDU provided methodological oversight and project management support throughout the process. The consensus panel, derived from GDG membership, played a significant role in developing the guideline through agreement on recommendations based on summaries of evidence.

The SME-TG comprised evidence review experts, all of whom had training in data analysis and evidence-based medicine. They performed systematic searches, appraisal, and synthesis of evidence for specific content. An SME-TG lead supervised group members who developed evidence summaries and recommendations and presented these to the Consensus Panel. Roles and functions are outlined in Table 1.

No patients were involved in any significant role in the development of this guideline. However, in advance of the publication of the Guidelines for the Public Health Management of Measles in Ireland (2025), the GDG engaged with the National Patient and Service User Forum, and a final draft copy was made available for feedback. This was completed in February 2025.

2.4. Conflict of Interest

To adequately moderate potential conflicts of interest and uphold standards of integrity, conduct, and concern for the public interest, all GDG members were required to complete a conflict-of-interest declaration form as a prerequisite for effective participation. The Chair and the RGDU screened and reviewed submissions by panel nominees for potential conflicts of interest. A policy for managing conflicts of interest was implemented in accordance with the HSE Code of Governance 2021 [32].

2.5. Sources of Evidence/Preparatory Research

The RGDU developed a detailed search strategy document (S5), and a systematic search was undertaken across Medline, CINAHL, PubMed, and the TRIP (Turning Research into Practice) database to identify potentially relevant resources. A hand search of Google and Google Scholar was also undertaken. As a consequence, four international guidelines were identified that met the inclusion criteria that had been set a priori and these included National measles guidelines (UKHSA); Measles for Healthcare Providers (Centers for Disease Control and Prevention); Health New Zealand (Ch 12 Measles) Communicable Disease Control Manual; Measles - Communicable Diseases Network Australia (CDNA) National Guidelines for Public Health Units.

¹AMRIC is a national team, established by the HSE, to address the challenges facing the Irish health service in dealing with healthcare-associated infection and antimicrobial resistance.

These guidelines were independently assessed for methodological rigour and transparency using the Appraisal of Guidelines for Research and Evaluation (AGREE II) Instrument [33]. Each guideline was appraised by two reviewers selected from a review group that comprised methodological expertise (RGDU) and subject matter expertise (GDG). High-quality guidelines were defined as those where the AGREE II score >70%.

A novel Stakeholder Engagement Form (S6) was employed to summarize the content agreed upon from the source guideline, and this was circulated to all members of the GDG to ensure that all relevant topics for the guideline had been addressed. Topics were identified as follows: Background, Public Health Management, and Specific Settings and Situations. Accordingly, the GDG decided to adopt and adapt the National measles guidelines (UKHSA).

In July 2024, upon the development of the three SME-TGs, it was evident that detailed training and education were required to enable coherence across each group to effectively implement the formal consensus methodology for the purposes of synthesizing evidence and expert opinions and formulating recommendations. The project team delivered bespoke training sessions based on a consensus-based recommendations protocol. This approach took account of the diverse previous experience of group members in both guideline development and the application of consensus methods.

2.6. Assessing Consensus

The formal consensus measures employed in this study involved multiple phases, including a modified e-Delphi round and two NGT meetings with the consensus panel. Adopting a modified e-Delphi approach meant that the project team established the relevant questions at the outset before submitting these to the consensus panel. [34] Primarily, the process involved extracting evidence summaries and recommendations from the source guideline and preparing decision logs for consideration by the consensus panel Fig. (2). The intention was to gather panellist input and agree on a conclusive list of recommendations for inclusion in a guideline for the public health management of measles in Ireland.

2.7. e-Delphi

Evidence summaries and recommendations were extracted from the source guideline for the agreed-upon topics. SME-TGs identified variations in practice upon analysis of the evidence summaries and recommendations from the source guidelines. The SME-TGs formulated a total of 36 statements around the topics included in the scope of the guideline. Consensus decision logs were prepared as MS Forms by each SME-TG for their area of expertise based on these statements. The aim at this juncture was to achieve consensus from all panel members on the statements presented. Statements were presented as either “updated content” or “new content,” with the rationale and evidence source for the modification stated. Panel members were presented with three response options to each statement: “Proposed content acceptable”, “Proposed content acceptable (with modifications)”, “Proposed content unacceptable”. Panellists were required to explain their responses if they found any of the statements to be either “acceptable with modifications” or “unacceptable”. The questionnaires with the list of statements are available as supplementary material (S7a, S7b, S7c).

Results from the e-Delphi round were returned individually by panel members. For ease of use, results were collated by the RGDU using MS Forms and Excel. A subsequent meeting of each SME-TG was held to formulate another iteration of statements based on the results collected from the panel. These statements were then presented to the consensus panel at the NGT meeting in October 2024.

2.7.1. Nominal Group Technique Meetings

Within this study, the purpose of the NGT meetings was to consider the updated statements that the participants had reviewed and commented on during the e-Delphi Round. The first meeting focused primarily on the statements that were the source of disagreement from Round 1, i.e., those that received responses of “Proposed content acceptable (with modifications)” or “Proposed content unacceptable.” Statements that had reached consensus “Proposed content acceptable” in the previous round were not included in subsequent rounds. The nominal group meeting was facilitated by the Chair of the GDG, an experienced NGT facilitator, which helped ensure that the process ran smoothly and that high-quality decisions were made. When all members had been given the opportunity to respond, there was a discussion to clarify, defend, or challenge the issues. All members of the group were given the opportunity to discuss all the issues they wished to. There was then an opportunity for each participant to re-rate the revised statements. Consensus was achieved when all panellists found the revised content “Acceptable,” and for the purpose of this study, it meant that every committee member found the proposed resolution acceptable, or at least lent it support, even if less than wholeheartedly. The final consensus was documented for all statements in the Decision Log.

2.8. Ethical Considerations

This study involves ‘normal and statutory public health work’ aimed at reviewing evidence and developing recommendations with members of the GDG. The Reference Research Ethics Committee Midlands and Corporate Services (HSE Dublin & Midlands) [35] confirmed that, as such, ethical approval was not required for this project.

3. RESULTS

3.2. Formal consensus phase 1: e-Delphi round – Chapter 2 (Public Health)

This e-Delphi round was coordinated to close on 11th October 2024 and followed an identical arrangement to that outlined above. Within this chapter, the SME-TG identified 19 draft recommendations, considering content adapted from the source guideline (11) and de novo content (8).

Once again, a response rate of 87% (n = 20) of panellists was recorded for Chapter 2. Three of the nineteen draft recommendations were accepted without modifications by all (100%) of the panellists who responded. Of the remaining sixteen draft recommendations, there were no instances where panellists indicated that proposed content was unacceptable. However, there were varying numbers (1-6) among panellists who indicated that the proposed content would be acceptable following modifications to the text. This was particularly applicable to the adapted draft recommendation on ‘Defining the infectious period’; and de novo content on ‘Defining exposure risk to vulnerable immunocompetent individuals’ e.g., infants and pregnant women.

3.3. Formal consensus phase 1: e-Delphi round – Chapter 3 (Specific Settings)

The final e-Delphi round was coordinated to close on October 22, 2024, and, once again, followed the same approach. Within this chapter, the SME-TG identified 11 draft recommendations, considering content adapted from the source guideline (7) and de novo content (4).

A response rate of 70% (n=16) was recorded for chapter 3. Two of the eleven draft recommendations were accepted without modification, and of the nine remaining recommendations, there were no instances where panelists indicated that the proposed content was unacceptable. Once again, there were varying numbers (1-4) among panellists who indicated that the proposed content would be acceptable following modifications to the text. This outcome was more focused on draft recommendations related to ‘Measles in settings for underserved populations’ (de novo content); and amended content referring to ‘Considerations for healthcare staff’; ‘Educational and Childcare settings’; ‘Air Travel’; and ‘Protecting hospitalised children from measles’. Consensus levels related to all draft recommendations are presented in Table 2, while a summary of the results is outlined in Table 3.

3.4. Formal Consensus Phase 2: Nominal Group Technique Meeting

The preliminary NGT meeting was convened on the 5th of November 2024. All members of the consensus panel were present, representing both regional and national stakeholders and expertise. The meeting was chaired by a public health consultant with previous experience in facilitating these groups. As was the case with the e-Delphi exercise, the processes underpinning the NGT meeting had been previously clarified for panel members by the RGDU. Nevertheless, the project team accepted the opportunity to summarise the approach once again, ensuring that all participants were familiar with the functions and purpose of the meeting.

Panel members were provided with copies of the decision logs for chapters one, two, and three, which had been amended following the conclusion of the e-Delphi phase. In the intervening period, the SME-TGs responsible for each individual chapter had reassembled to address the feedback received from the e-Delphi exercise. Consequently, evidence summaries were reviewed and actions focused on modifying proposed recommendations where necessary. These modified recommendations were now presented to the panel for review, discussion, and consensus.

Similar to Singh and colleagues (2013) [19], the benefit, in this case, of commencing a Delphi process in advance of the NGT meeting is that it created space for domains of consensus and non-consensus to be revealed. The impact of this was that the NGT meeting was able to effectively concentrate on the domains where there were differences of opinion and thus lacked overall consensus.

Panellists reviewed the proposed recommendations, with a particular focus on areas where previous variations in practice and divergence had been identified. The Chair moderated the ensuing discussion, ensuring that all panellists had an equal opportunity to submit an informed opinion. Following this, voting was facilitated on all draft recommendations where consensus remained outstanding following the e-Delphi phase. Consensus was established among all panellists for all draft recommendations within chapter one, except for that relating to ‘Transmission of primary measles’. In this case, it was agreed that the National Immunisation Advisory Committee (NIAC) should be requested to undertake a further review of evidence on this matter.

The process for advancing consensus was facilitated in an iterative manner for chapters 2 and 3 as well Fig. (3). As regards these chapters, the NGT process was followed for all draft recommendations and the requisite level for consensus (100%) was attained with the exceptions of items relating to ‘Immunocompetent vulnerable contacts: pregnant women’ (Chapter 2); ‘Considerations for healthcare staff’ (Chapter 3); ‘Air travel’ (chapter 3); ‘Measles in prisons and places of detention’ (chapter 3); ‘Protecting hospitalised children from measles’ (Chapter 3). The expert consensus panel determined that additional time was needed to review current evidence in respect of proposed recommendations impacting these settings and, if necessary, request further evidence synthesis and/or modifications.

3.5. Formal Consensus Phase 3: Nominal Group Technique Meeting

A second Nominal Group meeting was convened on 19^th December 2024. Once again, all consensus panellists were present, and the meeting was facilitated by the Public Health Consultant who had presided over phase 2. In the interim period, clarification had been received from NIAC on evidence impacting ‘Transmission of primary measles’. In addition, based on the feedback from the consensus panel during Phase 2, a review of evidence pertaining to other jurisdictions, and contextualization of the evidence within an Irish frame of reference, all outstanding draft recommendations received consensus among all panellists (100%) following the requisite modifications to the text. It was further agreed that the complete guideline document should be considered for approval by the national Health Protection Advisory Committee for Infectious Diseases in February 2025.