Sally Crook SiB programme Manager
In development circles ‘impact’ is an important, but sometimes still contested, concept.
This might be because from the thesaurus on my computer, the word ‘impact’ is given both a positive, but also a negative connotation. Positive: Influence. Negative: Crash.
As managers, clinicians and providers we need to focus on the positive! And avoid the crash, the shock. Knowing the true impact of your project can be hugely positive for your own project and organisation and its future work, but also for the whole eye care sector. But as the thesaurus suggests, impact can also be negative, often due to lack of an agreed strategy and clear budgeting at the start of the project cycle.
Donors and senior managers of programmes today frequently ask “What will be, or what was, the impact of this project?” To give some assessment, possibly in a final report or presentation, on the impact of a project you will need to undertake an evaluation.
This is because impact is more than just “counting up all your outcomes” – the metrics that were used in monitoring the project during its lifetime. Impact wants to say something more, something bigger.
Impact assessment needs to look back and review how the project was delivered, including to assess points such as how the project met equity and access issues, or how cost effective the project was in delivering both the planned, and any unplanned, outcomes; and to look forward, into how progress will be maintained (sustainability). This process of undertaking an impact assessment will enable the team to capture and share both the positive successes for this population, and also any key learning. Learning helps both understand what worked, and why, and also what did not work as expected, and why not. Remember, impact will always be moderated by local factors.
In health care the impact is felt at both the individual, patient level, and also at the public health, or population level. In eye care a clear impact is felt by the individual patient through the improvement or stabilisation of their vision after receiving one-off, or continual, eye care services. Their assessment of the intervention is important.
Or, as phrased in a recent CEHJ article “Are they happy? And if not, why not?” And for each patient this is important. But to judge the overall impact of a programme or project we also need to know if and how we are helping the targeted population in the most efficient and effective manner.
In order to measure impact of a project, you need to capture clear outcomes that can then be assessed alongside benchmarks (around quality, unit cost etc) for specific outputs. For some health interventions there are clear standards that need to be met by all service providers. These standards guide implementers and their managers and enable them to demonstrate that they are meeting universally accepted standards in their work.
In eye care: For cataract surgery it is necessary for the team to meet WHO targets for “good and satisfactory” surgical outcomes. For mass drug (including Vitamin A) distributions to be effective it is usually required to cover >80 or >90% of the target population and over several years. For training specific cadres of health staff there are both national and international standards that need to be met.
When there are agreed and accepted standards, or benchmarks, it is much easier to conduct impact evaluations at the end of a project. Assuming baselines and standards are known and agreed by the project staff, and good records have been kept.
But this is not the whole story – as any clinician or manager knows. You could have achieved good surgical outcomes, but by refusing to operate on very elderly patients or anyone suspected of having a co-morbidity. Population coverage targets could have been met by “drawing the line” to exclude the communities or parts of communities that are more remote or just too difficult to access (over the river, mobile, out of school, up the mountains….). Or, rather than train national staff, short term contract or expat staff could have been used to deliver specific project outputs. So this is where context is crucial, and agreement over the population to be covered (the denominator) and also how this project will meet their needs to agreed standards is important. A project can have high unit costs, but if the issues and problems to be addressed for a specific population are clearly stipulated, then these higher unit costs will be acceptable.
Another key variable in assessing impact is whether the outcomes are sustainable. Impact includes some assessment of service delivery into the future. Or, will everything just collapse once the project funding ends? Will equipment be maintained, will appropriate referrals still be made to the eye unit, will consumables be sourced on time, and will staff remain at this unit?
These strongly affect the future delivery of the activities, and hence the long term health impact of the project activities. And also the cost-effectiveness (and hence impact) of your project.
If an operating microscope has an expected life of x years, but you only use it for 3 years (i.e. only during the life of the project), and not in the future due to lack of equipment maintenance, surgical team not having correct skills, staff leaving the unit, or whatever reason, then the value for money (and hence impact) achieved by the funds immediately decreases. The local providers need to have been supported to further utilise their improved skills and fully utilise any equipment provided to have high impact for this population.
Impact is just a continuum of a logical project framework or results chain. If the project has been clearly planned, with clear, pre-defined outcomes from specific outputs then an impact assessment conducted at some point in the future can clearly assess how the project has contributed to meeting the needs of the population through improved and sustained outcomes in eye care. And that the outcomes meet the internationally or nationally agreed benchmarks mentioned above. ODIs recent “10 Things to Know About Evaluation” is a great synthesis of somewhere to start when you plan to assess the impact of your project.
A useful diagram to think about impact being the final link in the project’s results chain comes from a VfM project funded by DfID.
For a larger version of the above diagram, please click here.
As stated above, impact is closely related to the cost effectiveness and the cost efficiency of the use of the funds available in a project or programme – the ‘value for money’ around how the project was delivered. See the small arrows at the bottom of the schema above. The UK’s Department for International Development (DfID) defines Value for Money (VfM) as “maximising the impact of each pound spent to improve poor people’s lives” (DfID 2011). And this is also echoed in the UK National Audit Office definition, which defines VfM as being “the optimal use of resources to achieve intended actual outcomes”. The key point from these two definitions is to make the best use of available resources to achieve sustainable development outcomes to agreed standards. So impact and VfM are clearly interlinked.
Many managers worry that impact assessments, and especially VfM assessments, are all about donors reducing the unit cost – but if your project has addressed throughout its life (including the design stage) four key points: Problem Analysis, Strategy agreement, Actors and action, Performance evaluation. Then a final impact assessment can acknowledge the situation at the start and hence the investment and action that was required to develop services to deliver quality eye care for the population. So to measure true impact you do need to get beyond the VfM issues. Or as INTRAC phrases it,
“Impact therefore assesses the ‘So what?’ question: how has this project actually affected the lives of the people it aimed to support?”
SiB Programme Manager
 Robert Lindfield 2015 “Measuring the outcome of cataract surgery: the importance of the patient’s perspective” CEHJ 2015 Vol 28 Issue 89 pp12-13.
 INTRAC M&E Paper #7. Maureen O’Flynn Oct 2010. http://www.intrac.org/data/files/resources/695/Impact-Assessment-Understanding-and-Assessing-our-Contributions-to-Change.pdf