Strategic Health Sector Strategic Plan Evaluation

Executive Summary

Following the adoption of the Malawi National Strategic Plan 2015-2020, (NSP), the Government of Australia has commissioned an external consultant to develop a Measurement and Evaluation plan to accompany the revised NSP. The goal of this new M&E Plan is to guide the collection of strategic information for effective management of the HIV in the youth population, and to measure the intermediate strategic outcomes as well as the long-term impacts. The NSP (2015-2020) provides the frame of reference for the new M&E Plan and a program logic model provides a clearly illustrated pathway to achieving the short, medium and long-term goals.

This evaluation plan has been aligned to the timeframes established in the NSP and sets out to identify the activities and measurement indicators that will determine the progress towards reaching the targets and goals as set in the Millennium Development Goals. An efficient measurement and evaluation system is critical to the country’s HIV management response, it ensures that decisions are based on both qualitative and quantitative data, assists in policy creation and ensures accountability and transparency are achieved.

Key stakeholders have been identified and a communication plan established to ensure all stakeholders are consulted and kept informed as to the progress of the evaluation.

A combination of formative and summative evaluation and multiple sources of data are required to provide a comprehensive and accurate summary of the progress towards achievement of goals and to identify components that are working and what is not working. Some of the data will be collected through interviews, focus groups, health facility survey and review of existing population health census data.

The evaluation results will be used to make evidence-based decisions in relation to the allocation of resources, the cost-benefit of the related activities, and essentially to determine if the current activities are successfully meeting their objectives. This information will also inform future planning and program design and help funders to make decisions about the level of support they provide. Some of the potential stakeholder who will use information include, policy makers within MoH, advocacy groups, funders, NGOs, community youth-based institutions, service providers, health technology/information agencies and by students of Public Health.

Background Information

The Malawi Health Sector plan was formulated to guide the implementation of equitable, efficient and quality health services and to strengthen the overall healthcare system of Malawi. The Australian Government has agreed to provide the financial and technical assistance for developing a revised Health Sector Plan as a basis for an evaluation plan. The revised plan is based on a Population Health framework and identified the high rate of HIV/AIDS as a high priority health issue for Malawi. The review also identified the greatest risk factor associated with this health condition is unprotected sex or “unsafe sex”, which also contributes to multiple of other health issues such as high maternal mortality and mother to child transmitted conditions.

Malawi has made some progress in responding to its HIV epidemic, with positive results from the prevention of Mother to Child transmission. (UNAIDS, 2017) However, Malawi’s HIV/AIDS prevalence is still one the highest in the world with 9.2% of the adult population (15-49) living with the condition and 36,00 new HIV infections in Malawi in 2016. (UNAIDS, 2017)

Undertaking this evaluation will provide more information to understand the health issue and will provide crucial information to establish benchmarking data for future evaluations.

Goals and Objectives

A key element of the revised Health Sector plan was to identify the measures of success that could be used to determine progress towards achieving the planned goals and objectives. This evaluation plan is based on the priorities, overall goal, objectives and strategies as identified in the recent review of the Malawi Health Sector plan.

The key goal identified in the revised health sector plan was to reduce the prevalence of HIV/AIDs among the 15-24yr age population to 5% by 2020.

The main cause relating to the transmission and prevalence of HIV/AIDS in Malawi is unsafe sex practices in the young population.

Three objectives to address the cause were identified as:

Objective 1 – Increase the use of condoms within the 15-24yr old age group to 70% by 2022

Objective 2 – Increase the percentage of 15-24yr old age group that are tested for HIV/AIDSs to 80% by 2022

Objective 3 – Raise awareness of how to prevent HIV amongst school aged children

Short term goals – Formative Evaluation

• Enhanced perception of personal risk
• Increased knowledge of risk reduction strategies
• Increased knowledge of HIV transmission

Medium term goals – Formative evaluation

• Increased frequency in correct condom use
• Reduction in the number of unsafe sex interactions
• Improved social attitudes towards the use of condoms

Long term goals – Summative evaluation – (impact)

• Trends in HIV rates
• Sustained changes in societal norms
• Sustained changes in HIV risk related behaviours

Purpose of the evaluation

The purpose of the evaluation is to determine the degree to which the existing programs and activities are succeeding or failing to meet their objectives and to identify ways to improve the outcomes of these programs.

The results of the evaluation can be used by funders to make informed decisions programs worth and to understand the factors that lead to the success or failure of a program for future planning. 

There are many benefits to be gained by undertaking evaluation;

• Accountability and transparency
• Evidence based decision making relating to resources allocation, program design, implementation, etc
• Improved understanding of the health issue and of client needs
• Improved collaboration/consultation among the program team and with relevant stakeholders.

Parameters of the evaluation

Evaluation can vary greatly in scope, it can look at one activity within a program, or the complete program or a range of programs delivered through one agency or a series of programs delivered by multiple agencies. (Butler, 2008) The scope of this evaluation includes those activities specifically identified as part of the logic framework to address the prevalence of HIV in the Malawi youth. This evaluation plan does not include any indicators, outcomes or activities that sit outside the logic framework or seek to evaluate the Health System of Malawi at a national or at a global level. 

Governance

It is important for a clear governance structure to be in place for all evaluations, to monitor evaluation processes that are being undertaken within the parameters of the evaluation, to ensure compliance with data collection requirements such as privacy & confidentiality, timeliness of the evaluation and how the results are distributed. In this instance the MoH in partnership with the National Aids Committee will provide the appropriate guidance and approvals and resources to successfully complete the evaluation.

Despite the successes of the recent Health Data Collaboration approach in Malawi, fragmentation persists, frequently caused by development partners introducing systems that are duplicative or are siloed by programmatic area or geography. High-level leadership within the Ministry can help ensure better alignment. (Health data Collaborative, 2018)

Evaluation Design

Program Logic outcomes

Evaluation Questions

Based on the program logic the evaluation will attempt to answer the following questions;

• What outcomes have been achieved for the target group
• Are the outcomes sustainable and adaptable to further sites/locations?
• What improvements can be made to the program?
• How many of the target group received HIV education?
• How many of target group do not have access free condoms/education?
• What regions/areas have not met minimum condom delivery/provision targets
• What factors act as barriers to taking up safe sex practices?
• What communication/marketing approach has the greatest effect?

Evaluation components

Evaluation is most useful if performed in a logical sequence: first collecting information on input, process and output; then examining behavioural or immediate outcomes; and finally assessing disease and social effects. (impact)

Both input (the basic resources required in terms of human resources, funds, equipment and time) and output (the immediate improvement demonstrated through distribution of resources, trained staff, and service units delivered) are key elements of process evaluation. Process-oriented evaluation is carried out throughout the program implementation phase and will include different methodological approaches to measure awareness and assess service delivery, ranging from reviews of community education services, health records and regular reporting systems, interviews of service users, and voluntary counselling and testing [VCT] services) to quantitative population-based surveys to assess progress to goals or gaps in service.

Measuring health outputs and outcomes

An evaluation approach that uses multiple data collection methods, both quantitative and qualitative, is more likely to address diverse evaluation needs than is a more limited approach. All stages of evaluation need to be considered together to provide an overall picture of the program because no single data collection approach can supply all the information necessary to improve program performance or affect policy change. Multiple complementary evaluation approaches and multiple methodologies need to be applied to address different evaluation needs.

Population health data can be collected from reviews of existing information such as, annual health facility census, patient data from clinics, providers, MoH, NGO’s and other external agencies or community service providers.

Awareness can be measured through focus group discussion, short qualitative surveys, individual interviews and participant observations.

Evaluating the effectiveness of HIV/AIDS prevention programs will include quantitative measurements. These measurements will assess the extent to which the objectives of the program were achieved and attempt to answer the questions, “What outcomes were observed?,” “What do the outcomes mean?,” and “Does the program make a difference?” (Family Health International, 2013)

Cost-effectiveness analysis may be undertaken in order to determine a program’s effectiveness, by  comparing the costs and results of various activities, cost analyses and cost effectiveness estimates can help with resource allocation, setting priorities and program design. (NSW agency for Clinical Innovation, 2013)

Communication of findings

Stakeholders will have a vested interest in the results of the evaluation. (CDC, 2013) In general, stakeholders are those who are;

1. interested in the program and would use evaluation results, such as clients, community groups, and elected officials;
2. those who are involved in running the program, such as program staff, partners, management, the funding source, and coalition members; and
3. those who are served by the program, their families, or the general public. (ACI, 2013)

The key stakeholders should be included in the evaluation planning process, gaining their active support and participation, particularly for programs funded by external donors that rely on the host country for their own data collection activities. To increase the utilization of evaluation results, the design, planning, analysis and reporting of monitoring and evaluation should actively involve key stakeholders, such as district and national programme managers, policy-makers, community members and programme participants.

Data produced in a collaborative effort will have a better chance to be timely and of acceptable quality. Stakeholders are much more likely to buy into and support the evaluation if they are involved in the evaluation process from the beginning. (CDC, 2011)

National AIDS programmes, Ministries of Health and other agencies can work together to enhance  monitoring and evaluation and must commit to identifying sustainable resources if monitoring and evaluation is to be a routine programme function

Communication Plan

An outline for what information needs to be provided to who and when is just as important as what information needs to be collected from who and when. A communication plan outlining the method of communication, the purpose and the information to be provided forms would draw from the stakeholder analysis developed in the strategic plan.

Determining whether observed changes or findings in HIV incidence and prevalence are due to an intervention or a result of the natural progression of the epidemic is an important evaluation issue. This is relevant when evaluating behaviour or attitude changes in the face of growing numbers of people with AIDS-related illnesses because there is evidence that trends toward risk reduction will occur within close social circles. For example, having a family member or close friend with HIV/AIDS may influence the youth to either delay undertaking sexual relations or motivate those with non-regular sex partners to use condoms.

Evaluation timeframe & resources

The evaluation timeframe reflects the goals and objectives identified in the strategic plan. The program logic model outlines the short, medium and long-term goals and provides a guideline for the evaluation timeline. The overall program goal is to reduce the prevalence of HIV/AIDs among 15-24yr age group by 2020, measuring the impact of a program or summative (impact measurement) may take 2-5 years and will most likely be retrospective. Formative evaluation can be undertaken during the program planning and implementation stages can therefore take from two months up to two years.

In ideal situations, each kind of evaluation is carried out at the appropriate point along the program timeline to continually improve and increase the likelihood of achieving planned goals. Evaluation that is forward looking or prospective determines what activities should take place and evaluation that is backward thinking is retrospective – and attempts to understand what actually happened. (Butler, 2008)

The resources required to undertake this evaluation include the human resources, information technology resources, equipment and learning resources and marketing materials.

Conclusion

A practical, well-coordinated and strategic monitoring and evaluation plan is essential to minimizs the burden of data collection on country partners while maximizing the usefulness of monitoring and evaluation data for decision-making. The benefits to be gained from this evaluation plan include:

• It provides opportunities to develop integrated national and sector specific monitoring and evaluation stems to guide a national response to HIV and AIDS
• It assists in responding to international funding commitments and reporting requirements;
• It provides the platform for partnership, networking, and collaboration between national-level and local-level stakeholders in monitoring and evaluating national and local responses to HIV and AIDS.

Evaluating HIV/AIDS prevention and care programs is an ongoing challenge, but recognising its importance in improving current interventions may help to enhance the success of future initiatives

To minimize the data collection burden and maximize limited resources, monitoring and evaluation activities need to be well coordinated and utilize ongoing systems that collect and analyse data. 

With evidence-based and real-time data, the Malawi government through the Ministry of Health will be able to make informed decisions to improve population health outcomes and make measurable progress towards the goal of achieving Sustainable Development Goals.

References

• ACI, NSW Agency for Clinical Innovation, Understanding Program Evaluation, An ACI Framework. 2013,  www.aci.health.nsw.gov.au, accessed on 11.10.2018.
• Family Health International, Evaluating Programs for HIV/AIDs Prevention and Care in Developing Countries,
• Mwayin Kachapila, MoH Policy Brief, How should the Malawi Health Sector indicators be revised to align with the current monitoring and evaluation needs? July 2016.
• UNAIDS2017 Data Book, (accessed online 10/9/2018) http://www.unaids.org/en/resources/documents2017_data_book
• Round, R, Marshall, B & Horton, K 2005, Planning for effective health promotion evaluation, Victorian Government Department of Human Services, Melbourne.
• Developing an Effective Evaluation Plan. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; Division of Nutrition, Physical Activity, and Obesity, 2011.
• Health Data collaborative, Country Case Study – Malawi, 2018. Accessed online 12.10.2018) www.healthdatacollaborative.org/fileadmin/uploads/hdc/Documents/Country_documents/HDC_Country_Case_Study_Malawi.pdf)
• Butler, J, Ardal, S, Hohenadel, J, Olsen, D 2018. The Health Planners toolkit, Health System Intelligence project – Module 6 Evaluation, Ministry of Health, Ontario, 2008 accessed online 7/10/2018 www.ontla.on.ca/library/repository/mon/22000/283845.pdf