An Analytical-Predictive Model for Measuring the Efficiency and Effectiveness of Public Health Services

1. INTRODUCTION

Public health services are organised differently in the world, but their common task is improving populations’ health. Public health services are usually provided by institutes or agencies that organise professional activities for the improvement and protection of a population’s health as well as reducing mortality, premature deaths, and disabilities. The subject of this research is the prevention activities organised by the institutes of health services in the Republic of Croatia (hereinafter: Croatia). They are oriented to monitoring, analysis, and evaluation of the population’s health, which require goal setting, developing a planning system and implementing adequate measures. In other words, a proper performance measurement system, the role of which, besides monitoring and evaluation, should also be adjusted to communicate the extent to which various aspects of the health system meet the key objectives [1].

Today’s digital environment and the ‘Big Data Age’ provide new capabilities and organisational, managerial, and strategic benefits [2] that will contribute to a more accurate and comprehensive measurement system. These datasets that contain volumes of different unstructured, semi-structured and structured data are, besides other 'Vs', characterised by data from different sources, data accuracy, and reliability (veracity) that examine the cost-benefit of data collecting (value) [3, 4]. It has been acknowledged that Big Data analytics has an important influence on improving the efficiency and quality of health care delivery [5]. Manyika et al. [6], point out that properly applied data analytics helps cut costs by approximately $300 million annually in the health care industry. Additionally, it also helps improve lifestyle management. Accordingly, Wang et al. [2] point out the following potential operational benefits of big data analytics in health care: it improves workflow efficiency, monitors quality, improves costs and outcomes, and reduces the time for extraction of information from research studies on large databases.

The health service measurement system is very complex because of various data expressed through different measures. Due to a wide range of inputs and outputs, a comparison between units is sometimes problematic and even impossible. At the same time, since the introduction of the concept of ‘new public management’ to the public sector, the measurement of efficiency and effectiveness has become almost mandatory. Public health institutes are also under pressure to provide their services as efficiently as possible. This means achieving the highest quality while operating rationally. Due to sufficient evidence of inefficiency in the health care system across EU countries [7], the efficiency of health care services is often subject of research [8-13]. However, research related to the efficiency of preventive public health services, which is of great importance for society as a whole, is very scarce. Public health services in Croatia are carried out through various programmes and projects organised at the national level along with twenty county-level institutes. According to the Healthcare Act adopted in 2008 [14], the activities of public health institutes include: microbiology, ecology, epidemiology of communicable and chronic non-communicable diseases, promotion of health, school medicine, addiction prevention, mental health, and public health care. Some of the important roles of the institutes are the evaluation of the population’s health, health care needs, and the results of health care services, as well as the introduction of health programmes and their coordination at the national and international level.

All these activities require a developed measurement system for efficiency and effectiveness assessment. Data Envelopment Analysis (DEA) introduced by Charnes, Cooper, and Rhodes [15] is very often used in health care for measuring relative efficiency [16-26]. DEA is a mathematical programming method for evaluating the relative efficiency of different units operating in similar conditions and the conversion of inputs into outputs. It identifies the sources and the level of inefficiencies and shows how some of the inputs and outputs can be improved to become relatively efficient for an inefficient unit. DEA could be used together with the Balanced Scorecard (BSC) and some studies found great potential in their integration [27, 28], especially when quantitative and qualitative data are used to obtain a comprehensive performance and efficiency management system [29]. BSC has some advantages, like improvement of the management information system, emphasis on the relationship between quantitative and qualitative indicators, and relatively few selected measures that can be checked at any time [30]. The BSC could not identify inefficiency in the use of materials and human resources. However, in combination with DEA, limitations such as lack of possibility to select the best measures and insufficient dynamics for simultaneous control could be avoided.

Today, health systems are still in the initial stage of performance measurement and much more needs to be achieved in order to improve their effectiveness [1]. Only a unified and comprehensive system that provides an information base for monitoring the efficiency and effectiveness of the results and set goals, enables the use of contemporary tools and methods. Therefore, every healthcare system should have set financial and non-financial efficiency indicators that lead to greater effectiveness measured through the quality improvement of services provided, and thus better health care policies at the national level.

Combining DEA with BSC in this research aims to develop an analytical, but also a predictive model for purposes of effectiveness. The main goal of any public health service is long-term quality improvements, so that the primary measure is effectiveness of a combination of outputs and set outcomes. The conceptual model proposes an adequate measurement system for the assessment of efficiency and effectiveness in preventive health service institutes.

2. ADEQUACY OF THE MEASUREMENT SYSTEM FOR EFFICIENCY AND EFFECTIVENESS ASSESSMENT

The variety of programmes and projects carried out by institutes of public health in Croatia makes efficiency measurement complex. The problem arises due to the diversity of ‘input’, ‘output’ and ‘outcome’ expressions. Inputs are generally referred to as physical, human, and financial resources. In public sector organisations, most measures are based on an input-oriented perspective, usually expressed in terms of cost, budget, and stuffing tools [31]. Comparison can be made with similar types of organisations or units using specific measures such as sampling cost, cost of analysis, cost per programme, unit cost per employee, resources used per programme, etc. Any change in these performance measures reflects the ‘effectiveness’ with which the organisation uses its resources. However, this provides little information about the operational process within the organisation, like the total cost for a single medical analysis, distribution of costs between units, overhead cost assignment, etc.

Outputs in the public sector are oriented to the satisfaction of their beneficiaries and generally cannot be traded in the market. From the aspect of health services, outputs refer to all prevention activities expressed as various analyses, samples, programmes, treatments, vaccination, etc. They are easily measured in quantitative terms, such as: the number of samples taken, number of blood analyses, number of educational programmes or educational events held, number of obese children, price of some prevention programmes, etc. Organisations usually use the calculated output/input ratio to measure how efficiently an organisation converts its inputs into outputs. This provides very little information on the effectiveness or the value of these outputs.

The problem arises when results need to be measured qualitatively or when it is necessary to express ‘outcome’. Nearly all outcomes of public health prevention activities are visible in the long run. Their goals are related to quality improvements and satisfaction of their beneficiaries, considering that public health prevention outputs should be constructed as quality/effectiveness metrics. For example, by increasing the number of vaccinated children, some diseases like measles will diminish or disappear in the long run, or more frequent measurement of sea pollution will have an impact on the purity and quality of the sea. Thus, the ultimate goal of public health services is the quality improvement for the beneficiaries, so the first necessary step is to set up the desired target values that need to be achieved in the long run. This is only possible with the measurement of technical efficiency, i.e. establishment of whether the same effects could be achieved with fewer people, in a shorter period of time, with less material etc.

Efficiency measurement sometimes uses financial inputs aiming to help answer questions about whether the output could be produced less expensively; e.g., could we have the same laboratory analysis quality when the cost per unit is reduced. To meet the beneficiaries’ needs, ensure responsible management of resources and the staff, and efficiently organise business processes, a set of various efficiency and effectiveness measures should be developed and implemented.

Although some systems move faster than others in adopting performance indicators, many countries, even those with advanced data systems, have difficulties linking practical performance to outcomes due to limitations in data availability and their linking [32]. Like in all other public systems, efficiency measurement of public health systems is a challenging task that includes conceptual and technical problems. Some of the problems highlighted by Kattel et al. [33] that can be directly linked to the Croatian public health system are the wide range of beneficiaries and difficulties in defining goals, as well as a static measurement as opposed to dynamic processes. Besides, the database system, as a precondition for creating an adequate measuring system in today’s Big Data Era, has an important role. Regarding the Croatian public health system, this phenomenon is still characterised by three main factors that need to be improved: lack of control, information integrity, and transparency [34].

Comparability is another problem, because of the heterogeneity of public health services. The issue that arises in the evaluation of efficiency and effectiveness is the comparability of inputs and outputs. Comparison is possible when monetary inputs or outputs are used, such as revenue, total costs, material costs and using common quantitative measures like the number of employees, working hours, education hours, etc. Sometimes adjustments are necessary because of the technological process; e.g., a department takes blood for testing, but some blood tests are performed in another department. In that case, costs are allocated in accordance with the time needed for each activity or a method is applied to eliminate variable pricing or differential cost structure in health care institutes.

3. METHODS AND RESULTS

3.1. The Usefulness of the BSC DEA Model for Predictive Purposes

The advantages of the BSC model have been recognised in the health system for a long time. It is used in the health care system worldwide as a management tool that can help organisations effectively implement strategies [35], or develop practical conceptualisation, mainly for hospitals [36-39]. Regardless of the advantages, BSC could not identify inefficiency in the use of material and human resources. For that reason, its combination with DEA can help avoid these limitations.

Public health services in Croatia are managed through various programmes and projects organised at the national level along with twenty county-level institutes. This research was carried out at the Teaching Institute of Public Health in one of the Croatian counties. The primary tasks of the Institute are monitoring, analysis, and evaluation of the population’s health, as well as planning, proposing, and implementing measures for the preservation and enhancement of the population’s health. Development of a model which the management of the Institute will use for the research of performance and prediction analysis starts with DEA applicability and, later, with BSC development and implementation.

A detailed analysis of the process was conducted first, regarding the organisation and processes in each of the 12 departments encompassing total 48 units (DMUs). Inputs and outputs were analysed for each department, and the departments differ according to their orientation to the patients they treat, or to a broader population in which they conduct prevention activities (prevention of epidemics, the examination of production workers, or food distribution workers). Some departments monitor and analyse the environmental factors and are not in direct contact with the population, and some also conduct diagnostic tests. Therefore, both inputs and outputs are different, and timing is also different, i.e. some activities provide results immediately or in a short period of time (vaccination, microbiological analyses), and some programmes are conducted for several years (different prevention programmes and early detection of diseases). Thus, in this phase of the research, the easiest way to test DEA was using comparable quantitative and financial data. An input-oriented model (BCC) was used, which is more appropriate than CCR due to variable return to scale permission. The BCC model evaluates the efficiency of DMUo (o= 1, 2, ..., n) by solving the following linear programme [40-42]:

Moreover, we took the BCC model for the period of three years (2014-2016), because the current health policy fosters rationality through reduced expenditures, but the management also has more control over the inputs.

INPUTS:	OUTPUT:
- working hours      - number of employees      - material costs      - investment in equipment	- realised and collected revenues

Some departments, for example, Health Ecology, operate in the market; it was thus interesting to analyse payments through collected revenues. We found that, at the level of all 48 DMUs, a third was relatively efficient in revenue collection. The projection to the efficient frontier was determined for inefficient DMUs as well as types of return to scale (RTS - increasing, constant, and decreasing). Most DMUs had a growing return to scale, which means for the decision-making process, the inputs should be increased for possible achievement of greater revenues.

To confirm the applicability of DEA to each of the departments, we chose the Department of Health Ecology that covers 12 DMUs and provides health prevention services regarding different environmental factors (air, food, water, soil…). We continued applying the BCC model and selecting inputs and outputs that were reliable and comparable for all units within the department. In this case, we used the following inputs and outputs:

INPUTS:	OUTPUTS:
- total costs      - number of samples	- total revenues      - number of analyses

Inputs and outputs were highly correlated i.e., the correlation between total costs and total revenues was approximately 0.97, between the number of samples and the total revenues, it was approximately 0.70, between total costs and number of analyses, approximately 0.61, while between the number of samples and number of analyses, it was approximately 0.53. With the calculation of return, we established that only four units out of 12 had an increasing return, but only three units (Biological Monitoring and Exposure Unit, Ecotoxicology Unit, and Waste Control and Waste Water Unit) with an increasing scale showed the possibility to improve efficiency by scaling up their activities. A projection that enables movement of an inefficient unit to an efficient frontier was also calculated.

By testing DEA at the level of the institute and the chosen Department of Ecology, we confirmed that DEA is applicable with the selected common inputs and outputs. The next step was to create BSC for the entire Institute and its departments. We started with the mission and vision and selected four core perspectives. The first and the crucial one is the perspective of beneficiaries and other stakeholders (local community, suppliers, investors...) because of whom the institute exists. Continuous investments in the staff will encourage improvements and innovation, thus increasing internal efficiency and ensuring more financial funds for strengthening of the quality of services. Fig. (1) represents the strategic map with the perspectives and strategic objectives of the institute.

Based on the strategic map of the institute, all departments have developed their own BSC and strategy map based on the results obtained by DEA. For example, the Department of Health Ecology develops the map shown in Table 1.

The strategic map of the Department of Health Ecology contains in total eleven goals under four perspectives. Hence, it is not entirely identical to the strategic map of the Institute of Public Health. It is evident that the strategic map of the Department of Health Ecology does not have the “U4” user perspective goal from the Institute’s maps, which refers to the expansion of the public health service market. Hence, although the Institute’s departments work together in order to fulfill the common set of goals, their mission and vision, each department has its own particular function and therefore, not all can, nor they should have the same goals, which also includes a strategic map. They develop their maps on the basis of the set institute map, but by adopting the indicators that were considered the most important and feasible to follow. More precisely, the differences between the departments’ maps lie in the “U” and “L” perspective, while the “F” and “P” are the same for all departments. For example, the Department of School and University Medicine, whose primary goal is health care protection of school children, the youth and students, and the Department of Social Medicine, whose primary goal is to safeguard and promote the health of all residents and visitors. They will not have the “L2” objective: “modernisation of equipment and investment in supporting infrastructure (learning perspective goal). Their focus in the learning perspective will be on the development of innovative models and on the enhancement of external and internal education. By contrast, the Department of Epidemiology and Health Ecology, whose main task is testing various samples, will incorporate it due to the importance of equipment modernisation. Hence, deciding which goals and related indicators to include was a long-term process and subject to group meetings with the leaders and members of each department (in this case, the Department of Health Ecology, the Department of Controlling and the Board of Directors of the Institute).

Table 1.

Mission: To preserve and promote the health of all residents and visitors in the Primorsko-Goranska County through the protection of all environmental components. Vision: Improving health and sustainable development through monitoring and analysis of environmental factors.
PER.	GOALS		MEASURES
U	U1	Increase customer satisfaction through the quality of services provided	• customer satisfaction rating (survey; 1-5) • number of current complaints / number in the previous year • number of services provided / funds planned • number of services provided / previous period
	U2	Increase the total number of environmental components’ testing	• number of current tests / number in the previous year • number of current tests / planned number of tests • number of references / number of elaborates
	U3	Increase the number of new types of services	• number of new services / total number of services
F	F1	Provide sufficient financial resources to provide quality services	• average value (in HRK) per team of the Croatian Health Insurance Fund (HZZO) • the share of income of the Croatian Health Insurance Fund (HZZO) in total revenues
	F2	Increase revenues from services on the market	• share of market-realised revenues in total revenues • amount of invoiced services / total number of employees in the provision of services
	F3	Ensure financial sustainability	• collected revenues / expenses • collected revenues / invoiced revenues
P	P1	Certify, accredit, and integrate new management systems	• number of implemented management systems / number of planned management systems (ISO/IEC 17025, ISO 9001, ISO 14001) • number of accredited methods / total number of methods
	P2	Improve the efficiency of internal processes with new organisational and IT solutions	• number of services provided / effective hours of work • maintenance cost method / total revenues from that method • number of justified complaints / total number of complaints about service provided
L	L1	Adoption and dissemination of knowledge	• number of specialised educations / total number of employees • number of planned educations / number of realised educations • number of internal educations / number of external educations
	L2	Modernisation of equipment and investment in supporting infrastructure	• value of new equipment / total revenues
	L3	Increasing innovative solutions	• number of innovations (methods, procedures, analysis, processes ...) • number of published professional and scientific papers

(Source: Own compilations)

3.2. The Conceptual Analytical-predictive Model for Efficiency and Effectiveness Measurement (APE)

BSC is created using DEA results about relative efficiency and benchmark units, which help the heads of the departments to set achievable short-term and long-term objectives. According to the strategic map created, heads of each department within the Institute could measure the efficiency of daily or monthly resource use . By measuring the efficiency with a different set of ratios for each perspective, the head of each department could easily get reports and analyse results and predict the degree of achieved targets . For example, the Department of Ecology identified DMUs that could achieve the same level with less costs and fewer employees. Projections to the effective frontier were used as target size in the BSC and prompted appropriate actions that will have an impact on the quality of public health (for example, department reorganisation). Therefore, the so-called APE model for efficiency and effectiveness measurement is presented in Fig. (2).

The model will be explained based on the Department of Health Ecology which covers 12 DMUs and provides prevention services for people’s health regarding environmental factors. The predictive analytical efficiency model consists of several phases.

In the first phase, we analysed relative efficiency using DEA for the period from 2014 to 2016. The selected inputs are total costs, number of samples and outputs, the number of analyses and total revenues; all highly correlated. The objective was to analyse which of the 12 DMUs was the efficient one in terms of the number of analyses and revenue realisation. They all perform in the same environment with appropriate equipment and technology. Using the DEA, we identified that three DMUs were efficient during the entire three-year period: for quality control of outdoor air, for the control of drinking water and natural water, and for the control of physical environmental factors.

In the second phase, BSC was created for the Department of Health Ecology with four perspectives. DEA results are used as a benchmark or targets from the perspectives of beneficiaries and other stakeholders, and from the point of view of financial management. Heads of the departments set targets until the end of 2020, attempting to measure operational efficiency on a monthly and yearly basis.

The third phase encompasses the analysis of the results of 2017 to discover whether the set targets (for example, increase in the number of analyses for 10 per cent and increase in revenues for 5 per cent) would be achieved by 2020. The results will be checked again using the DEA and the targets will be corrected if necessary. The BSC DEA model uses information technology and programmes that enable everyday evidence. Heads of departments can correct operational goals and predict the degree of fulfillment of the set long-term goals. This helps control operational efficiency and ensures long-term goals. Measurement of public health is important for the overall quality of the health care system and any improvements in operational efficiency will lead to long-term effectiveness.

CONCLUSION

Today’s orientation to digitalisation will undoubtedly have an impact on the methods, instruments, skills, and organisations used in the health care system. The health care system, in general, is oriented to meeting the expectations of its beneficiaries, aiming to ensure a healthy life for everyone. In this paper, we presented a conceptual model for efficiency measuring in an institute of public health. Their mission is preservation and promotion of health of citizens and visitors through the more efficient supply of health services. Therefore, efficiency measurement is crucial for achieving their mission and vision and, consequently for long-term improvement of the quality of life of all citizens of the county.

The predictive analytical model proposed in this paper is Balanced Scorecard combined with Data Envelopment Analysis to measure relative and technical efficiency as well as strategic effectiveness. The model used DEA as a benchmark for targets set in each perspective within the BSC. Operational efficiency is measured by indicators chosen for each objective along with initiatives. The results need to be correlated with target sizes to determine the degree of reaching the final goal set for several years in advance. Implementation of the APE in the institutes of public health will contribute to the improvement of analysis, forecast, and optimisation of all their activities and achievement of higher decision-making quality.

However, we are aware that the implementation of the model in a real case may have limitations. They are primarily related to the development of the information system as well as human resource ability and willingness to change. Secondly, there are some limitations in the methods used. DEA is a useful tool for assessing efficiency and identifying needs for improvement in similar organisations or units. In our case, when applied to a single institution, there is a limit in the number of DMUs, which also limits the number of inputs and outputs. So far, many studies have confirmed the usefulness of BSC, but some have criticised it. However, in our case, the management supports the BSC measurement system, and implementation problems may arise from different public health policy solutions (board member or organisational changes, sources of financing, etc.)

REFERENCES