Santé et Services sociaux Québec.
rss
Previous page Adjust text size

Brief analytical summaries or syntheses #44

Health system performance comparison:

an agenda for policy, information and research

Summary

This WHO policy brief calls for more international comparisons of health system performance. It describes the methodology of international comparative studies, key domain comparators, and cites the benefits of international comparison in policy influence and as a source of evidence.

Background

This policy brief seeks to summarize the current “state of the art” of health system comparison, identifying data and methodological issues and exploring the current interface between evidence and practice. It also draws out the priorities for future work on performance comparison, in the development of measurement instruments, analytic methodology, and assessment of evidence on performance. It concludes by presenting key lessons and future priorities policy-makers should take into account.

Analysis and results

Comparing key domains of performance:

  • Population health measures often take a broad perspective, which captures the effect of many determinants of health beyond the delivery of health care. This broad perspective can be attractive from a political point of view because it draws attention to the importance of many sectors in determining health outcomes. However it also creates major methodological challenges in seeking to attribute changes in health to any particular actions. A narrower perspective, such as the concept of avoidable mortality, focuses on measures that can more easily be attributed to health care, and which are therefore more amenable to immediate health policy.
  • All population health indicators suffer from a number of methodological problems, which need to be addressed in order to make international comparisons more meaningful. Some of the main issues involve availability and coding of data, particularly of data on cause of death that are difficult to compare between countries and over time.
  • Direct indicators of the contribution of health services to health status are available in the form of health service quality measures, such as standardized hospital mortality rates and numerous disease-specific health outcome measures, such as mortality rates, adverse events and complications. Far less prevalent are broader outcome measures in areas such as disabilities and discomfort.
  • While existing health service measures offer some indicators of the performance of individual organizations, international comparison is complicated by different organizational settings and reporting conventions, even after suitable adjustment for case-mix and other contextual circumstances.
  • Although comparative indicators on inequality of health and equity in access to health care are available at both European and non-European level, equity indicators derived from existing projects and datasets may be misleading for policy-makers due to limitations in the availability and comparability of data.
  • The many aspects of financial protection have yielded some important indicators, such as the incidence of catastrophic expenditure. However it has proved difficult to develop a single indicator capturing the full extent to which people are financially protected from health shocks. Currently, measures of the incidence and magnitude of households’ direct payments for health care form the basis of metrics for financial protection assessment and system comparisons.
  • There is still lack of clarity as to what dimensions should be included in the domain of responsiveness, which embraces concepts such as respect, confidentiality and prompt attention. This uncertainty leads to the measurement of different areas using different tools with different weights, domains and indicators that are difficult to summarize and compare.
  • Efficiency indicators serve as a summary measure of the extent to which the inputs to the health system, in the form of expenditures and other resources, are used wisely to secure the goals of the health system. Almost all efficiency indicators are constructed as a ratio of inputs to outputs, offering an indication of the extent to which resources have been wasted along the production pathway.
  • In measuring efficiency, a fundamental challenge is the assignment of inputs and associated costs to specific health system activities, often relying on arbitrary accounting rules or other questionable assignments. In principle, the inputs used should be directly and fully aligned with the output under scrutiny.

Conclusion

If undertaken carefully, health system performance comparison offers a powerful resource for identifying weaknesses and suggesting relevant reforms. The progress that has been achieved is impressive, both in the scope of areas for which comparable international data on health are now available and in the degree to which comparability has been improved. However, the science of international comparison is at a developmental stage. Policy-makers therefore need to be made aware of both the strengths and limitations of health system comparison.

Future directions

The key requirements for creating comparable indicators that address the needs of policy-makers are: appropriate methods of summarizing complex information; a narrative that picks out the key issues and uncertainties; a diagnosis of why the reported variations are arising; and an assessment of the implications for policy action.

Implications and recommendations

Policy issue

  • International health system performance comparisons have the potential to provide a rich source of evidence as well as policy influence.
  • Country comparisons that are not conducted with properly validated measures and unbiased policy interpretations may prompt adverse policy impacts and so caution is required in the selection of indicators, the methodologies used, and the interpretations made.

Lessons from international comparisons

  • Definitions of performance indicators should be clear and consistent, and fit into a clear conceptual framework.
  • The metrics used in international comparison should enjoy widespread acceptance, and be defined in unambiguous terms that are consistent with most countries’ data collection systems.
  • In order to draw meaningful comparisons and understand the drivers of differences in measures between systems it is usually necessary to adjust for variations in the demographic, social, cultural and economic circumstances of nations.
  • In defining the boundaries of the health system it is important to be aware of the benefits of choosing both narrow and wide boundaries.
  • Performance measurement evaluates the extent to which a health system meets its key objectives. However progress in the development of data collection techniques in the different dimensions of health performance is variable.

Source

Health system performance comparison: an agenda for policy, information and research