RESEARCH ARTICLE
Improving COVID-19 Vaccine Uptake in Saskatchewan, Canada: A Developmental Evaluation Approach
Maryam Yasinian1, Tracey Carr1, Jason Vanstone2, Amir Azizian2, Patrick Falastein3, Gary Groot1, *
Article Information
Identifiers and Pagination:
Year: 2024Volume: 17
E-location ID: e18749445288367
Publisher ID: e18749445288367
DOI: 10.2174/0118749445288367240311101720
Article History:
Received Date: 19/10/2023Revision Received Date: 20/02/2024
Acceptance Date: 28/02/2024
Electronic publication date: 21/03/2024
Collection year: 2024
open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: https://creativecommons.org/licenses/by/4.0/legalcode. This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Abstract
Background
The Developmental Evaluation of a COVID-19 vaccination program was an early response to assess a complex emergent mass vaccination program to support learning and adaptation.
Objective
The primary objective of a multi-disciplinary team of researcher-evaluators was to facilitate organizational learning among key stakeholders to improve decision-making and increase vaccine uptake in Saskatchewan, Canada.
Methods
Aligned with the Developmental Evaluation approach, data collection was rooted in adjustment and flexibility to meet the evolving needs of the vaccination program. Data were primarily collected using meeting observations and program documentation. As the program progressed, the data collection was adjusted, and two surveys were conducted targeting COVID-19 vaccine recipients and vaccine immunizers. Data were analyzed iteratively in consultation with stakeholders.
Results
Nine feedback reports were generated over a nine-month evaluation period. Seven reports highlighted meeting observation results that revealed the program issues, probable causes, and implications. The evolving issues ranged from vaccine shortage, delay, and supply fluctuation to inter-organizational miscommunication and vaccine hesitancy. Two reports were produced from survey findings to delve into the persistent issue of vaccine hesitancy.
Conclusion
Effective solutions to complex issues of Saskatchewan’s COVID-19 mass immunization require a systems approach based on new ways of thinking and collective decision-making.
1. INTRODUCTION
When the coronavirus disease (COVID-19) resulted in a worldwide pandemic [1], the production of effective vaccines and rapid immunization became a public health priority to control virus transmission, morbidity, and mortality [2]. After the approval of the first COVID-19 vaccine, the largest vaccination campaign in human history was initiated, and healthcare systems began COVID-19 vaccine distribution. Given COVID-19 immuni- zation was massive in scope and complex in nature, global leaders were confronted with unprecedented resource and distribution challenges [3]. In the context of limited supplies and ongoing uncertainty, rapid real-time evaluation of vaccination implementation was required to optimize the allocation of vaccines and to achieve the goal of herd immunity.
Experiences from previous outbreaks have demons- trated that reliable and timely information is vital for the success of vaccination implementation strategies [4, 5]. For instance, lessons learned from the Ebola epidemic, a disease that was declared a Public Health Emergency of International Concern [6], confirmed that the availability of vaccines alone is not sufficient to increase uptake [7, 8]. Rather, accurate information, timely evaluation, and tailored implementation strategies should constitute the foundations of Ebola vaccination programs. Lack of effective strategies has hindered national and inter- national efforts in Ebola eradication despite the investment of millions of dollars [5, 7]. Poorly designed vaccination plans against Ebola had undesirable outcomes such as the exclusion of the most vulnerable populations (e.g., pregnant women) [6], misunderstandings of side effects [4], vaccine hesitancy and refusal [4], public mistrust in vaccine deployment, eligibility, and prioritization plans [4], ineffective messaging and lack of community engagement [7], heavy toll on affected countries’ economies and their social fabric [5], and re-emergence of multiple Ebola outbreaks [7]. The key lessons learned from outbreaks such as Ebola have important implications for COVID-19 vaccination programs in terms of preparation, timely evaluation, clarification of accountability, and determination of vaccination needs.
In the context of the COVID-19 pandemic and surges of highly transmissible variants, the production of timely knowledge becomes critical for healthcare systems to identify gaps in immunization and adjust plans to optimize coverage (Alberta Health Services, 2021). For example, in Canada, the national (e.g., the National Advisory Committee on Immunization (NACI)) and provincial/ territorial advisory groups (e.g., Provincial Immunization Technical Group (PITGs)) synthesize available evidence and provide ongoing recommendations on multiple aspects of immunization and COVID-19 vaccines (e.g., vaccination of specific populations, vaccine safety and efficacy, hesitancy) [9-11]. Despite these knowledge synthesis and translation efforts, there is a lack of real-time evaluation of COVID-19 vaccination programs.
In response to the need for rapid assessment of the COVID-19 vaccination program in Saskatchewan, a Developmental Evaluation team was assembled to provide timely feedback to the Saskatchewan Health Authority (SHA) about its implementation throughout the province. In this paper, we outline our Developmental Evaluation approach to this process by first describing the fundamentals of a Developmental Evaluation and then delineating our rationale, methods, and findings. The goal of the evaluation was to facilitate organizational learning and quality improvement in the vaccination process by (1) providing rapid real-time production of feedback to facilitate COVID-19 vaccine-related decision-making; (2) identifying issues and their probable causes as well as their implications for the vaccination program; and (3) offering recommendations for continuous learning, adaptation, and innovation.
2. MATERIALS AND METHODS
2.1. Setting
Saskatchewan is a province in Western Canada with a population of 1,221,439 [12]. The Saskatchewan health- care system relies on the work and partnership of two key organizations, the Ministry of Health (MoH) and the SHA, that are accountable for the province’s healthcare vision, direction, and delivery (Government of Saskatchewan, 2020). The MoH operates as the legislative body that provides healthcare planning and oversight, while the SHA is responsible for health service provision [13, 14].
Similarly, the MoH and the SHA shared the responsibility for launching a province-wide COVID-19 immunization program by performing different but interconnected roles in strategy development and the vaccine delivery plan. The MoH had the overall responsibility for leading the vaccine campaign, setting priorities, and supporting the involved stakeholders, including the SHA, in the implementation of the vaccination program. The SHA was responsible for operational aspects of vaccine delivery, logistics, and administration plans in accordance with the MoH direction and provincial guidance. In the SHA, the vaccination program was co-led by two chiefs, an Executive Director and an SHA-affiliated physician with extensive experience in public health.
Using a phased approach, the first COVID-19 vaccine was administered in December 2020 based on priority populations, and as of November 2021, all residents aged five and older were eligible for vaccination. The key immunization milestones for Saskatchewan’s delivery plan were [14]:
• Pilot Phase (December 15, 2020): the first COVID-19 vaccines were administered to a targeted population of priority healthcare workers.
• Phase 1 (December 22, 2020): the first phase of vaccination was initiated in which the target population was expanded to long-term care and personal care home residents and their staff as well as priority healthcare workers.
• Phase 2 (March 18, 2021): the second phase began with the general population becoming eligible for vaccination based on age groups and vulnerability.
• Expanded Eligibility (May 20, 2021): as part of phase 2, all residents ages 12 and older became eligible for vaccination.
• Booster Program (September 7, 2021): as part of phase 2, the booster program was launched to administer third vaccine doses.
• Pediatric Program (November 23, 2021): as part of phase 2, all residents ages 5 and older became eligible for vaccination.
2.2. Evaluation Approach
As first conceptualized by Patton [15, 16], Develop- mental Evaluation “supports innovation development to guide adaptation to emergent and dynamic realities in complex environments” [17]. Therefore, the application of Developmental Evaluation is well-suited for emergent evolving programs that are complex, dynamic, and unpredictable [15, 18]. This approach is particularly beneficial for evaluating and promoting healthcare programs that are in their early stages of implementation, when organizational learning is essential and when changes must be integrated into new programs [19].
In this study, it was determined that Developmental Evaluation was a well-suited approach to evaluating the vaccination program for four key reasons. First, innovation and adaptation were required daily in the COVID-19 vaccination program. Public health authorities were adapting to ongoing changes (e.g., roles, supplies, timelines), and the program clearly required innovation in every aspect, from strategies (e.g., eligibility plans, mobile vaccination sites, pop-up clinics, incentives) to human resource allocation.
Second, the province-wide program served as an emergent response as opposed to an existing program. When we began the evaluation, the vaccination program was in the second month of implementation. The program leaders were implementing the largest provincial vaccination program and grappling with evolving issues of the first phase of immunization, such as maximizing vaccine uptake while minimizing inefficiency. The program’s activities were also frequently changing, and its unpredictable nature necessitated a flexible and dynamic evaluation design that was open to adjustment and redirection, direct engagement, and learning.
Third, the vaccination program was one of the most complex SHA programs and contended with complicated gaps and high demands (e.g., ambiguity, scale, supply chain, logistics, distribution, quality assurance, manage- ment, community engagement, and involvement of diverse stakeholders). Consequently, rapid real-time feedback from the program team and external evaluators was crucial to address the complexity and ambiguity of the program.
Finally, the existing relationships between key stakeholders were a determinant for the selection of the Developmental Evaluation approach. The evaluation lead was an SHA physician leader who also led the Saskatchewan COVID-19 support team. Due to a strong relationship with the vaccine chiefs, it was possible to provide the SHA Emergency Operations Center (EOC) with reliable and timely COVID-19 evidence [20].
To conduct this Developmental Evaluation, the core evaluation team engaged in four principal concurrent and commonly practiced activities [21]: orientation, data collection, sensemaking and data analysis, and dissemination.
2.3. Team Composition
The core Developmental Evaluation team was comprised of six external evaluator-researchers from the College of Medicine, University of Saskatchewan (USASK) (G.G., M.Y., T.C.), internal evaluators in the SHA (J.V., A.A.), and a director from the provincial health quality council (P.F.) who were assigned to the evaluation roles as well as two internal key stakeholders who had lead roles in the Saskatchewan COVID-19 vaccination program. The external evaluators (G.G., M.Y., T.C. from USASK and P.F. from HQC) brought diverse expertise and outside perspectives, which were critical for this complicated, time-sensitive project. The internal evaluators (G.G., J.V., A.A.), on the other hand, played a key role in assessing and capturing the root cause of vaccination program barriers using their firsthand knowledge of SHA operations. The Developmental Evaluation team was built upon the pre-existing relationships between the lead evaluator (G.G.), who has a cross-appointment with the SHA and the University of Saskatchewan, and the two SHA vaccine chiefs. This prior relationship provided a foundation for initial trust, mutual understanding, and strong collaboration. As the project proceeded, the relationship was strengthened when the goals, scope, roles, and nature of the project were clarified. The lead evaluator’s affiliation with the SHA also helped the project to obtain a better understanding of the SHA organizational structure, which was needed to both navigate decision-making at a provincial level and interpret data effectively. The competencies of Developmental Evaluation team members are important considerations, as they provide credibility and impact users’ judgments of the entire evaluation process [16, 18].
Interpretation and data analysis were conducted collectively by the six evaluators. As the key users of this evaluation, the vaccine chiefs’ involvement was crucial in tying the evaluation to primary users’ goals and priorities for the ever-changing COVID-19 vaccination program (e.g., to policies, practices, resources). This collaboration was intended to empower key actors in the COVID-19 vaccination program using active learning and the evaluative thinking required to consider and design new approaches to vaccination implementation.
The evaluation lead clarified to the SHA stakeholders that the team’s role as developmental evaluators was not the assessment of the vaccination program. Rather, the team would facilitate learning that could be adjusted based on the program’s needs over time. Because Developmental Evaluation requires real-time observation to document the program and provide timely feedback, M.Y., J.V. and A.A. attended different SHA vaccination rollout meetings to initiate the data collection. The SHA vaccine chiefs supported this approach and agreed with the collective work.
| Meeting Title – Frequency | Attendees – Number of Attendees | Meeting Purpose | Evaluator(s) |
|---|---|---|---|
| SHA Core COVID- 19 Vaccine Meeting – Daily |
SHA vaccine leads and supporting staff (e.g., communications, logistics and distributions, quality and safety) – 15, approximately |
Give updates on their team’s work. Indicate upcoming work. Discuss challenges or barriers to escalate and solutions. |
M.Y. |
| Task Team for Optimization, Distribution, & Logistics – Daily | SHA distribution & logistics team (e.g., representatives from quality and safety, clinical excellence) – 7, approximately |
Give updates on their work. Indicate upcoming work. Discuss challenges or barriers to escalate and solutions. |
M.Y. J.V. A.A. |
| Immunization Readiness 2.0 Huddle – Daily | SHA vaccine clinics Quality Improvement (QI) leads and supporting QI staff, Executive Director (Quality and Safety), Director (Clinical Excellence) – 15, approximately |
Vaccine clinics QI leads of each geographic region (North, Rural, Regina, Saskatoon) give updates on their team’s work. Indicate upcoming work and daily targets. Discuss challenges or barriers to escalate and solutions. |
M.Y. J.V. A.A. |
| IHICC/ EOC – Daily |
SHA IHICC leads and their physician dyads, EOC commander and supporting staff, Vaccine chiefs, Attendees from various supporting departments and portfolios (e.g., communications, logistics and distributions, human resources, security, etc.) – 20, approximately |
The implementation teams of each geographic region (North, Rural, Regina, Saskatoon) give updates on their work. Discuss challenges or barriers to escalate and solutions. Discuss team resiliency and well-being. |
J.V. A.A. |
2.4. Data Collection
According to the Tri-Council Policy Statement (TCPS2), Article 2.5, program evaluation studies that have the sole purpose of assessment and quality improvement are eligible for ethics exemption [22]. This study was a quality improvement project and, therefore, ethically exempt; however, it was performed according to national (TCPS2) and international (Helsinki Declaration) guidelines. We commenced the Developmental Evaluation in February 2021, in the early days of the program when the province was implementing Phase 1, and concluded the evaluation in October 2021, when booster doses were being delivered. Our data collection was flexible to meet the evolving needs of the vaccination program. We primarily relied on observation and documentation of the program.
As it progressed, the data collection methods were adjusted based on the program's needs, and two surveys were conducted.
2.4.1. Observation and Meeting Documentation
When the evaluation was initiated in February 2021, direct observation was identified as the best approach to provide timely feedback. The vaccine chiefs selected four online SHA COVID-19 vaccination staff meetings for observation (see Table 1), including the SHA Core COVID-19 Vaccine Meeting, Task Team for Optimization, Distribution, and Logistics, and Immunization Readiness 2.0 Huddle. Beginning in May 2021, the evaluation team participated in a fourth meeting, the Integrated Health Incident Command Center(s)(IHICC)/EOC Vaccine Immunization, to gain more insight into the vaccination program. M.Y., J.V., and A.A. were assigned to the meetings as observers for data collection.
At the beginning of the evaluation, the observer team took detailed notes to document the vaccination program development from a broad lens. This method allowed the team to understand better the program group dynamics and structure as well as help the team determine where to focus their attention. Nevertheless, we realized that the sheer volume of data from the wide array of subjects discussed in the meetings required a learning template to identify what kinds of information to collect. After discussions with the SHA chiefs, we focused our attention on documenting high-level issues and barriers as well as their probable causes and implications. Some meetings (e.g., Task Team for Optimization, Distribution, and Logistics) in which the program staff voluntarily shared their in-depth reflections and experiences with the observer team in an informal setting had a smaller number of attendees. Additionally, the evaluators conducted one-on-one meetings with the program staff to facilitate reflection and elaboration of their experiences when more information was needed.
2.4.2. Survey
As the vaccination program progressed and vaccines became readily available across the province, the evaluation team and the stakeholders identified vaccine hesitancy to be a persistent obstacle that hindered updates.
We recognized that meeting observation reports were insufficient to evaluate the issue of hesitancy, its probable causes, and the implications for the program. In Saskatchewan, vaccine hesitancy was determined to be a multifaceted issue affected by socio-demographic variables such as geographic location, economic status, and ethnicity [23]. Therefore, the evaluation team collaborated with the Vaccine Hesitancy Working Group (relevant stakeholders from the SHA, Northern Inter-Tribal Health Authority (NITHA), Indigenous Services Canada (ISC), First Nations and Métis Health Services (FNMHS), and the MoH) to survey vaccine recipients and immunizers.
Two brief open-ended surveys, one aimed at vaccine recipients and the other at immunizers, were approved by the vaccine chiefs. Developed using the Research Electronic Database Capture tool (REDCap), the surveys were distributed by the SHA between August 24 and September 1, 2021. Vaccine recipients were invited by immunizers in vaccine clinics to complete either an electronic or paper survey that gathered demographic information (i.e., age, gender, location, and vaccine dose number). They included two questions: (1) What made you decide to get a COVID- 19 vaccine today? and (2) What do you think would get people who have not been vaccinated to get a COVID-19 vaccination? Immunizers participated in a separate electronic survey, which was distributed via email and inquired about their geographic region and asked: What do you think is needed to get vaccine hesitant individuals to get a COVID-19 vaccination? Participants were also asked to provide up to three reasons why, in their experience, vaccine-hesitant individuals are not getting a COVID-19 vaccine.
2.5. Data Analysis
2.5.1. Observational Data
The data from the SHA meetings were analyzed and interpreted collectively by the core evaluation team on a weekly basis and discussed with the SHA vaccine chiefs bi-weekly. In weekly meetings, the core evaluation team explored patterns, themes, and lessons learned from the observers’ meeting notes. The discussions were intended to uncover the most significant obstacles at the current phase of vaccination as well as their probable causes and implications for program development and efficacy. Because the lead evaluator attended external vaccination program meetings, he brought additional information to the meetings, which was incorporated into the reports. After each weekly meeting, two team members (M.Y. and T.C.) produced a feedback report based on the team’s analysis. The feedback report was then shared with the core evaluation team, who reviewed it prior to report finalization.
2.5.2. Survey Data
Survey responses were analyzed by two members from the core evaluation team ((J.V., A.A.) and four data analysts from the SHA. The open-ended responses were coded thematically based on Crawshaw et al.’s analysis method and then categorized into the barriers and enablers to vaccine uptake identified by vaccine recipients and immunizers [24]. J.V. and A.A. then reported the survey results to the core evaluation team and the vaccine chiefs for reflective discussions. The results and reflections were compiled by M.Y. and T.C. into two feedback reports, which were refined by the core evaluation team, finalized for distribution, and published [25].
2.5.3. Dissemination
The finalized feedback reports were submitted to the vaccine chiefs by the evaluation team lead, which resulted in an invitation to present to and become a member of an MoH-led vaccine hesitancy working group. The objective was to support the vaccination program development and improvement through the conduct of high-quality feedback production and rapid circulation.
| Report Number, Evaluation Time Window & Vaccine Phase | Evaluation Team Findings |
|---|---|
| Report 1 |
Issue: Lack of alignment between the MoH and senior SHA leadership regarding key strategic decisions. Summary: When the MoH has made abrupt changes in direction, it has led to miscommunication, confusion, stress, and potential disillusionment by people delivering the vaccination program. The delivery teams and their leadership are continuing to endure the multiple challenges of an already complex and unprecedented implementation of COVID-19 vaccines. Ongoing asymmetry between the SHA leadership and the MoH could compromise the efficacy of the COVID-19 vaccine roll-out and the well-being of the SHA personnel. If these issues are not addressed, effective management of the COVID-19 pandemic could be jeopardized. |
| February 25 to March 12 | |
| Phase 1: targeting priority population | |
| Report 2 |
Issue: SHA and MoH communication issues with regard to vaccine safety and efficacy. Summary: As vaccine hesitancy is a growing concern, consistent and transparent messaging around vaccine safety and efficacy is paramount. An adequately resourced and coordinated SHA and MoH communication plan is needed to address this hesitancy and inform the public about the possibility of supply fluctuations and the impact of different vaccines on their health. Providing the public with transparent information in advance regarding the side effects and possibility of supply fluctuations will increase satisfaction with the rollout and increase vaccine uptake. |
| March 13 to April 15 | |
| Phase 2: targeting the general population | |
| Report 3 |
Issue: Implementation of pharmacy pilot vaccine distribution without the SHA oversight. Summary: Because the MoH has unilaterally implemented the pharmacy rollout, the SHA has no oversight and therefore no ability to maintain an overarching coherent vaccination plan. |
| April 15 to May 7 | |
| Phase 2: targeting the general population | |
| Report 4 |
Issue: Vaccine distribution in schools and pharmacies introduces additional variation to the implementation process. Summary: While the involvement of different players is important to the success of vaccine implementation, it is critical for all the players to communicate and follow a consistent strategy plan. Otherwise, the efficacy of the vaccine delivery could be negatively impacted. |
| May 13 to May 19 | |
| Phase 2: targeting the general population | |
| Report 5 |
Issue: Second dose hesitancy, confusion, and frustration for the public. Summary: Rapid vaccination requires dissemination of reliable information, a consistently transparent immunization system, and evidence-based reopening plans to prevent a fourth wave of the pandemic and further public dissatisfaction. |
| May 19 to June 07 | |
| Phase 2: residents age 12+ became eligible for the first dose of vaccination | |
| Report 6 |
Issue: Ongoing COVID-19 vaccine hesitancy despite the availability of vaccine services. Summary: COVID-19 vaccine hesitancy can set Saskatchewan back in immunization goals and reopening plans if appropriate interventions, based on reliable evidence and the specific needs of each community, are not implemented. The current issue could also be an opportunity to understand public perceptions and learn how to tackle hesitancy barriers to a mass immunization program. Recommendation: Surveying hesitant subgroups by location could provide the necessary data to focus the strategy for the remainder of the summer. |
| June 07 to June 17 | |
| Phase 2: residents age 12+ became eligible for first dose of vaccination | |
| Report 7 |
Issue: Despite vaccine supply exceeding demand in SK, the planned pace of COVID-19 vaccination has slowed. Summary: Vaccine allocation to Saskatchewan has increased and has exceeded demand. Distribution models require adjustment, and hesitancy needs to be addressed by evidence-based, context-specific interventions. We have already developed and presented the COVID-19 Vaccine Uptake survey, which could offer data to better focus the strategy for the remainder of the summer. |
| June 17 to July 07 | |
| Phase 2: residents age 12+ became eligible for a second dose of vaccination |
3. RESULTS
Over nine months (February 2021 - October 2021), the core evaluation team provided nine feedback reports to the SHA vaccine chiefs. In total, 229 meeting notes were collected by the observer team, which resulted in the generation of seven feedback reports. The survey results led to the production of two feedback reports.
3.1. Observations and Meeting Documentation
Each observation feedback report presented three evaluation findings, including the (1) most significant issue(s) with the vaccination program during the reporting period, (2) probable causes of issues, and (3) implications for the program. Table 2 presents the report number, reporting period evaluated, vaccine phase of each report, and key evaluation findings (issue identified and report summary). Since the program was evolving, the reports were not aligned with specific reporting periods. Nevertheless, the team’s support was ongoing. It resulted in a minimum of one report per month (in March, April, and July 2021) and a maximum of two reports (in May and June 2021) contingent upon program needs.
3.2. Survey Results
The survey report summaries are presented in Table 3. The two reports were provided in October 2021, a month after the data collection was complete. Reports were created in response to evaluation feedback findings from May 19 to July 7, 2021, the vaccine chiefs’ requests, and survey findings that Saskatchewan had the lowest vaccination rate for the eligible population of all provinces [26]. Due to the importance of the issue, the evaluation team lead directly presented survey results to the MoH in addition to SHA vaccine chiefs. At this point, the Saskatchewan vaccination program had reached 78% of the eligible population (age 12+), and booster doses were available to those who were eligible. Thus, the evaluation team and stakeholders determined it was reasonable to conclude the Developmental Evaluation.
| Report Number & Survey Questions | Summary of Findings |
|---|---|
| Report 9 (vaccine recipients) | While there were common responses across demographics for Question 1 (e.g., To be able to travel; Positive attitudes/high perceived benefit of |
| Question 1: What made you decide to get a COVID-19 vaccine today? Question 2: What do you think would get people who haven't been vaccinated to get a COVID-19 vaccination? |
COVID-19 vaccines) and Question 2 (e.g., Mandates and restrictions; Having access to and trust in reputable information sources), we observed some demographic-related associations with vaccine uptake and recommendations of participants. For instance, older people were more likely to get the vaccine for travel purposes and ask for restrictions, whereas younger people were more likely to receive their vaccines due to mandates and recognize personal freedom as an approach to convince others to get their vaccines. These findings may reflect that individuals have different perspectives, depending on their demographic characteristics, that should be addressed accordingly. This data could help the government and policymakers to more effectively target their practices and messaging around the COVID-19 vaccination program. |
| Report 10 (immunizers) | There are prevalent common themes in immunizers’ responses for Question 1 (e.g., Having access to and trust in reputable information sources; Mandates and restrictions) and Question 2 (e.g., Media issues, including coercion, negative influence, oversaturation, scare tactics, conflicting information, misinformation; Concerns about COVID-19 vaccine necessity from different regions. For instance, having access to reliable information and media issues in terms of coercion, negative influence, oversaturation, scare tactics, conflicting information, and misinformation were ranked as top themes in response to both questions. Both themes highlight the significance of reliable sources of COVID-19 information for vaccine promotion and uptake. Considering each individual region, we observed order differences for one theme or across different themes. This could provide useful information for policymakers to address vaccine hesitancy based on the more specific needs of each region. |
4. DISCUSSION
The evaluation results (i.e., periodic reports) identified the multiple challenges that program stakeholders experienced during the vaccination program. Although the stakeholders had a general awareness of key issues, the evaluation offered critical insight into the intricacies of these complex challenges. Moreover, government recognition and subsequent adaptations and responses emerging from evaluation results validated stakeholders’ experiences. At program onset, the primary issues were vaccine shortage, delay in vaccine delivery, and supply fluctuation.
As the program progressed, the qualitative evaluation findings revealed that vaccine hesitancy became the greatest challenge to healthcare officials, who struggled with low vaccination rates despite an abundance of vaccine supply. The survey of vaccine recipients and immunizers identified reasons for vaccine uptake according to demographics. Older recipients were likely to become vaccinated to be able to travel, while younger recipients. Tended to report vaccine mandate compliance. The survey findings suggested more targeted vaccination promotion campaigns according to the motivations of the audience.
Another key finding was that collective and cooperative leadership and effective communication between the vaccination program’s regulatory authorities (SHA and MoH) were hindered by independent work within and between different portfolios of each organization (Table 2). These dynamics related to organizational structure are not uncommon in provincial healthcare systems and because health authorities do not always have the final say in policy decisions, the operationalization of change can be limited. During the Saskatchewan vaccination rollout, the conflicting priorities of the MoH and SHA made implementation burdensome for the SHA, which was frequently required to pivot (Table 2- Reports 1-4). For example, in Report 3, we noted that the MoH decided to include pharmacies in the rollout without adequate consultation with the SHA.
Similar issues have been reported for previous pandemics’ mass immunization programs (e.g., H1N1 influenza virus and Ebola disease). Congruently, in these cases, control efforts were disrupted by the initial scarcity or absence of supplies, lack of coordination and communication between multiple partners, vaccine hesitancy, and misperceptions about an unfamiliar disease [4, 27, 28]. The similarity between our findings and the gaps identified during previous pandemics has important implications for governments who aim to address these challenges.
Specifically, our results suggest that investment in public health infrastructure, a coordinated communicative task force, and the use of transparent, fact-based communication strategies can have significant effects.
A major strength of this Developmental Evaluation is that it aligns with other national and provincial initiatives in Canada (e.g., NACI, PITGs, the authors-led evidence support team in Saskatchewan) that aim to provide rapid real-time knowledge to identify gaps in immunization and adjust plans to optimize coverage [9, 11, 29]. However, to the best of our knowledge, none of these initiatives provided real-time commentary or evaluative feedback about vaccination program implementation. While the other initiatives relied upon literature reviews and evidence synthesis to provide up-to-date recommendations about coronavirus and COVID-19 vaccines, this Developmental Evaluation was uniquely committed to real-time evaluation of a vaccination program implementation utilizing ongoing learning, data collection, reflection, and analysis. In general, the key contribution of Developmental Evaluation is to generate knowledge about a program that will meaningfully inform decision-making and improve implementation [17]. Aligned with a systems-thinking approach, the evaluation involved internal and external evaluator-researchers, as well as the key actors of the vaccination program in learning and feedback production.
Using a Developmental Evaluation methodology worked well for the evaluation of vaccination implementation in the Saskatchewan context. While other evaluation methodologies, such as the WHO intra-action review (IAR), were available, the DE approach in the present study enabled us to employ methods that were suitable for our context and relied on our relationships with stakeholders [29]. There were similarities to an IAR in that we summarized strengths and challenges and used our reports to advocate for implementation improvement. However, our methods were ongoing and adaptive according to stakeholder needs.
For future emergency vaccination activity in Saskatchewan, Canada, or other countries, this evaluation method, where evaluators are closely embedded with decision-makers, has the potential to inform programming and improve resource allocation. The effectiveness of the method might depend on the quality of the relationships between evaluators and decision-makers [19]. In our case, having internal and external evaluator-researchers was essential to rapid communication and uptake of evaluation findings.
4.1. Limitations
Although the evaluation team took steps to identify and mitigate potential issues during the evaluation, four limitations were identified. The first limitation was the evaluation team composition. Although the SHA stakeholders were an information source, the team lacked the much-needed engagement of the MoH, as it was a key policymaker in the COVID-19 vaccination program. However, a collective and cooperative leadership model between the MoH and the SHA was not fully developed during the vaccination program. Consequently, SHA participation did not guarantee the flow of evaluation findings between the two organizations. This may have limited the evaluation’s impact on the advancement of emergent vaccination policies and practices. Likewise, team composition may have been a source of bias due to the involvement of only high-level stakeholders. However, this concern was mitigated by embedding two SHA employees within the core evaluation team and involving them in the vaccination program implementation.
Second, the rate of change and constant evolution of the COVID-19 vaccination program presented difficulties. Specifically, some challenges did not result in feedback reports because observations and identified issues were resolved quickly and/or replaced with new challenges requiring immediate attention. To address this challenge, we limited the scope of the evaluation to high-level barriers that directly affected the development of the vaccination program.
Third, securing the ongoing involvement of stakeholders in reflection and data analysis was a challenge due to their demanding schedules. Although the bi-weekly stakeholder meetings regularly took place, feedback reports were sometimes produced prior to stakeholder validation at subsequent meetings. That said, stakeholder validation of feedback reports is not a criterion that aligns with the purpose of a Developmental Evaluation.
Finally, the daily evolution of the vaccine program required real-time evaluation feedback. To offset this challenge and enhance the validity of findings, we relied on the expertise and knowledge of both internal and external evaluators who were actively engaged in generating evidence related to COVID-19 [20, 30]. Their knowledge of COVID-19 enriched their understanding of the COVID-19 vaccination program’s context and functionality. Additionally, the full-time work of some team members and the collection of daily observations ensured the validity of data collection and allowed for in-depth data analysis.
CONCLUSION
This Developmental Evaluation was implemented to provide Saskatchewan vaccination program decision-makers with the benefits of real-time evaluative knowledge about gaps and viable ways to adapt their operations to facilitate vaccine uptake. Despite challenges with the absence of some influential regulatory organi- zations (MoH), inconsistent stakeholder engagement in data analysis, and constant evolution of the vaccination program, this Developmental Evaluation provided the SHA with two valuable types of knowledge: (1) issues or barriers to vaccination program development, their probable causes, and implications; and (2) behind-the-scene organizational challenges related to decision-making, effective communication, and collaboration that could have otherwise went unvalidated without real-time evaluation. Our results present a strong message to Saskatchewan and other similar healthcare systems, namely, that effective solutions to complex health problems such as COVID-19 mass immunization require a collaborative systems approach involving diverse, multi-stakeholder networks.
LIST OF ABBREVIATIONS
| COVID-19 | = Coronavirus Disease |
| NACI | = National Advisory Committee on Immunization |
| PITGs | = Provincial Immunization Technical Group |
| SHA | = Saskatchewan Health Authority |
| EOC | = Emergency Operations Center |
ETHICAL STATEMENT
This study was a quality improvement project and, therefore, ethically exempt; however, it was performed according to national (TCPS2) and international (Helsinki Declaration) guidelines.
CONSENT FOR PUBLICATION
Not applicable.
AVAILABILITY OF DATA AND MATERIALS
The data and supportive information are available within the article.
FUNDING
None.
CONFLICT OF INTEREST
The authors declare no conflict of interest, financial or otherwise.
ACKNOWLEDGEMENTS
The Developmental Evaluation team gratefully acknowledges the support from the Saskatchewan Health Authority and the data analysts from the Clinical Excellence team. The authors would like to acknowledge Sheila Anderson, Dr. Tania Diener, and Collin Hartness for their support of this study. This research was part of the employment of the first author at the University of Saskatchewan and Saskatchewan Health Authority.
REFERENCES
| [1] | World Health Organization. WHO director-general’s opening remarks at the media briefing on COVID-19— 11 March 2020 2020. Available from: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020 |
| [2] | Dhama K, Sharun K, Tiwari R, et al. COVID-19 vaccine hesitancy – reasons and solutions to achieve a successful global vaccination campaign to tackle the ongoing pandemic. Hum Vaccin Immunother 2021; 17(10): 3495-9. |
| [3] | Abecassis A. Five priorities for universal COVID-19 vaccination. Lancet 2021; 398(10297): 285-6. |
| [4] | Carter S, Mobula L, Samaha H, Ahuka SM. Community engagement and vaccinations during the Ebola outbreak in Democratic Republic of Congo 2020. |
| [5] | Moon S, Sridhar D, Pate MA, et al. Will ebola change the game? ten essential reforms before the next pandemic. The report of the Harvard-LSHTM independent panel on the global response to ebola. Lancet 2015; 386(10009): 2204-21. |
| [6] | Gupta SB, Coller BA, Feinberg M. Unprecedented pace and partnerships: The story of and lessons learned from one Ebola vaccine program. Expert Rev Vaccines 2018; 17(10): 913-23. |
| [7] | Bausch DG. The need for a new strategy for Ebola vaccination. Nat Med 2021; 27(4): 580-1. |
| [8] | Galvani AP, Ndeffo-Mbah ML, Wenzel N, Childs JE. Ebola vaccination: If not now, when? Ann Intern Med 2014; 161(10): 749-50. |
| [9] | Alberta Health Services. 2021.https://www.albertahealthservices.ca/topics/Page17074.aspx |
| [10] | Government of Canada. Scientific advisory group COVID-19 recommendations. 2021. Available from: https://www.canada.ca/en/public-health/services/immunization/national-advisory-committee-on- immunization-naci/methods-process.html (Accessed on: July 11, 2023). |
| [11] | Public Health Agency of Canada. Process for incorporating economic evidence into federal vaccine recommendations: National Advisory Committee on Immunization (NACI) 2021. 2021. Available from: https://www.canada.ca/en/public-health/services/immunization/national-advisory-committee-on- immunization-naci/methods-process/incorporating-economic-evidence-federal-vaccine- recommendations.html (Accessed on: July 11, 2023). |
| [12] | Government of Saskatchewan. Saskatchewan’s dashboard - population. 2023. Available from: https://dashboard.saskatchewan.ca/people-community/people/population (Accessed on: July 11, 2023). |
| [13] | Understanding the Health Care System. 2020. Available from: https://www.saskatchewan.ca/residents/health/understanding-the-health-care-system (Accessed on: July 11, 2023). |
| [14] | Government of Saskatchewan. COVID-19 immunization delivery plan. 2021. Available from: https://pubsaskdev.blob.core.windows.net/pubsask-prod/130160/COVID%252BVaccine%252BDelivery%252BPlan%252B-%252BOctober%252B2021%252BUpdate.pdf |
| [15] | Patton MQ. Developmental evaluation. Eval Pract 1994; 15(3): 311-9. |
| [16] | Patton MQ. What is essential in developmental evaluation? on integrity, fidelity, adultery, abstinence, impotence, long-term commitment, integrity, and sensitivity in implementing evaluation models. Am J Eval 2016; 37(2): 250-65. |
| [17] | Patton MQ. Developmental evaluation: Applying complexity concepts to enhance innovation and use 2011. |
| [18] | Gamble JAA. A Developmental Evaluation Primer 2008. |
| [19] | Fagen MC, Redman SD, Stacks J, et al. Developmental evaluation. Health Promot Pract 2011; 12(5): 645-50. |
| [20] | Azizian AR, Carr T, Muhajarine N, et al. Developing a patient-oriented realist evaluation for COVID-19 vaccine implementation in Saskatchewan: A methodologic framework. CMAJ Open 2021; 9(4): E1034-9. |
| [21] | Dozois . A practitioner’s guide to developmental evaluation. 2010. Available from: https://mcconnellfoundation.ca/report/practitioners-guide-developmental-evaluation/ (Accessed on: July 11, 2023). |
| [22] | Government of Canada IAP on RE. TCPS 2 (2018) – Chapter 2: Scope and approach. 2018.https://ethics.gc.ca/eng/tcps2-eptc2_2018_chapter2-chapitre2.html |
| [23] | Muhajarine N, Adeyinka DA, McCutcheon J, Green KL, Fahlman M, Kallio N. COVID-19 vaccine hesitancy and refusal and associated factors in an adult population in Saskatchewan, Canada: Evidence from predictive modelling. PLoS One 2021; 16(11): e0259513. |
| [24] | Crawshaw AF, Farah Y, Deal A, et al. Defining the determinants of vaccine uptake and undervaccination in migrant populations in Europe to improve routine and COVID-19 vaccine uptake: A systematic review. Lancet Infect Dis 2022; 22(9): e254-66. |
| [25] | Vanstone JR, Azizian AR, Berry W, et al. Identifying barriers and enablers to vaccine uptake from immunizers and individuals receiving a COVID-19 vaccine in Saskatchewan. Open Public Health J 2022; 15(1): e187494452208012. |
| [26] | COVID-19 vaccination tracker 2022. 2022. Available from: https://covid19.trackvaccines.org/ |
| [27] | Burkardt HJ. Pandemic H1N1 2009 (‘swine flu’): Diagnostic and other challenges. Expert Rev Mol Diagn 2011; 11(1): 35-40. |
| [28] | World Health Organization. Factors that contributed to undetected spread of the Ebola virus and impeded rapid containment. 2015. Available from: https://www.who.int/news-room/spotlight/one-year-into-the-ebola-epidemic/factors-that-contributed-to-undetected-spread-of-the-ebola-virus-and-impeded-rapid-containment |
| [29] | World Health Organization. Guidance for conducting a country COVID-19 intra-action review (IAR). 2020. Available from: https://www.who.int/publications/i/item/WHO-2019-nCoV-Country_IAR-2020.1 |
| [30] | Groot G, Baer S, Badea A, et al. Developing a rapid evidence response to COVID‐19: The collaborative approach of Saskatchewan, Canada. Learn Health Syst 2022; 6(1): e10280. |
