The impact assessment of the project results can be based on the analysis of specific evaluation concerns, the choice of which will determine the scope of each evaluation. The basic information to be compiled in an independent evaluation usually covers three main areas of enquiry: design, delivery and performance. Design and delivery issues refer to factors affecting project results that manifest themselves during implementation. Performance issues are more directly concerned with the effects of project results. The main issues to be addressed in each of these areas of enquiry are listed below. (Endnote 4):
Validity of design. A good project design guides the work to be done, facilitates the monitoring of project implementation and provides a solid base for the evaluation of project performance. It also establishes the commitment of the project partners which increases the project’s chances of success.
In addressing the question of design validity, evaluations assess whether the original project design and its revision(s): (a) were logical and stated in unambiguous terms; (b) concentrated on the identified problems and needs and spelled out the strategy to be followed for solving them, (c) described the target groups and how the benefits would accrue to them, (d) placed the project in the appropriate institutional framework and defined the roles and responsibilities of the main project partners, (e) set out clear objectives and corresponding indicators of achievement to measure the changes to be brought about by the project; (f) described the main outputs, activities and inputs needed to achieve the objectives, (g) stated valid assumptions about the major external factors affecting project implementation and performance, and (h) indicated the required prior obligations to be fulfilled by the main partners.
Delivery process. The assessment of the delivery process focuses on the factors affecting project execution to determine if the project has kept on the right track towards the achievement of its objectives through the various operational stages.
Evaluations, therefore, are concerned with: (a) reviewing how the inputs through activities were transformed into outputs; (b) analysing the quantity, quality and timeliness of the outputs delivered; (c) examining whether the technical and administrative guidance and support provided by the project staff, partner organisations and the relevant ILO units (the Multidisciplinary Technical Advisory Teams (MDTs), field offices and headquarters) were adequate; (d) ascertaining the extent to which external and/or internal factors have influenced the project results; (e) determining whether the project’s response to such factors was adequate; and (f) recommending, if appropriate, what adjustments and follow-up actions are required to increase the chances of success.
Performance. When assessing the performance of a project, evaluations look beyond implementation factors and focus on the effects and impact of the work done. The outcome of this assessment determines whether or not the project has achieved or is likely to achieve its objectives.
The core evaluation concerns to assess project performance are described and illustrated in the chart below.
Core evaluation concerns
Relevance examines the usefulness of the project’s results in solving the identified problems and meeting the needs of the target group(s). The analysis ascertains whether the project continues to make sense and identifies any changes that may have occurred in its context during implementation. The initial problems and needs may no longer exist. New problems and needs may have emerged as a result of political, economic, social and other factors, or even because of the project’s activities. Ultimately, the analysis determines whether the objectives are still valid or should be reformulated.
Effectiveness assesses the extent to which the project has achieved its objectives and reached its target group(s). The analysis determines if the expected results have been or will be accomplished and, if not, whether the statement of objectives should be modified (mid-term evaluation) or the project extended (terminal evaluation). This assessment is facilitated if the objectives and indicators of achievement adequately describe the desired situation at the end of the project. The changes brought about by the project are ascertained by comparing information regarding the pre-project situation (of the beneficiaries) with the existing situation at the time of the evaluation.
Efficiency assesses the results obtained in relation to the expenditure incurred and resources used by the project during a given period of time. The analysis focuses on the relationship between the project’s inputs, including personnel, consultants, travel, training, equipment and miscellaneous costs, and the quality and quantity of the outputs produced and delivered. It ascertains whether there was adequate justification for the expenditure incurred and examines whether the resources were spent as economically as possible. This assessment should also determine whether the actions by the various partners were complementary and identify alternative strategies to deliver more and better outputs with the available inputs.
Sustainability ascertains the extent to which the project’s results have had or are likely to have lasting effects after the termination of the project and the withdrawal of external resources. The factors affecting sustainability are examined on the basis of the priority assigned to the project by the direct recipients and/or intended beneficiaries. Their readiness to continue supporting or carrying out specific activities, or even replicate the activities in other regions or sectors of the country, is particularly relevant. The analysis also assesses the availability of local management, financial and human resources that would be needed to maintain the project’s results in the long run.
Causality examines the factors or events that have affected the project’s results. If the inputs needed to carry out the planned activities and deliver the expected outputs were available on time, the project’s implementation and performance would be successful. If, on the other hand, there were significant deviations from the planned schedules, the analysis would determine the reasons for such changes. The assessment should also analyse the effect of other factors such as technical, administrative or managerial constraints, inadequate inputs, failed commitment by project partners, insufficient funds, a faulty assumption or the effect of an unexpected external factor.
Unanticipated effects are identified during the evaluation to ascertain if a project is having any significant unforeseen positive or negative effects. Once identified, appropriate action can be taken to enhance or mitigate them for a greater overall impact.
Alternative strategies to solving the identified problems or needs are assessed, particularly if the original strategy turns out to be inappropriate. Evaluations examine whether alternative approaches might have had greater impact or might have been more cost-effective. This analysis is especially valuable when follow- up activities are being planned.
Not all of the above evaluation concerns have to be examined in every evaluation. The final choice will depend on the purpose of each evaluation. However, it should be clear that the larger the number of concerns, the more comprehensive (or less partial) the assessment of the project impact will be. If one or more of these concerns are assessed in isolation from the other ones, the evaluation results can be biased or misleading (or both) (Endnote 5). To ensure an adequate impact assessment in all cases, the ILO has, therefore, adopted a set of the evaluation concerns which should be addressed, as a minimum, in the evaluation of its activities. These were chosen from the above core concerns and are the following: relevance, effectiveness, efficiency and sustainability. (Endnote 6)
Special concerns. In addition to the above core aspects for the assessment of the project’s performance, an evaluation also identifies and examines any anticipated or unanticipated effects on issues of special concern to the ILO, namely (a) the application of International Labour Standards, (b) the promotion of equality between men and women in employment, and (c) the protection of the environment for sustainable development. Although these are special concerns for any project, they might not have been taken into account in the planning and design stages of a particular project; hence, explicit references to these concerns are not always found in all project documents. An independent evaluation is, therefore, a useful tool to rectify such a shortcoming, and should ascertain whether the project under review has addressed them or whether it should address them in the future. (Endnote 7)
Endnote 4: A list of suggested questions to be addressed in assessing the main evaluation concerns can be found in Annex 4.
Endnote 5: If, for instance, only the achievement of the objectives is assessed (effectiveness), the evaluators may find that the project has achieved its objectives, but no information will be available on whether the needs which justified the project still exist (relevance), or whether the resources allocated were used
economically (efficiency), or whether the results will be sustained once external assistance has been withdrawn (sustainability). A project may have achieved
its objectives (effective) but become less useful (irrelevant), if the problem or the needs change during the project's life-time. And vice versa, a project may have continued to make sense (relevant), but failed to reach the target groups (ineffective). Likewise, a project may be effective and relevant, but at an unjustifiably high cost (inefficient),
or its results will not have lasting effects (unsustainable). Obviously, the recommendation of an evaluation would vary and correspond to the situation found
and assessed in each case.
Endnote 6: See the ILO's Guide to the Monitoring, Evaluation and Reporting System (MERS), (Revised), 1996.
Endnote 7: See the Guidelines for the integration of gender issues into the design, monitoring and evaluation of ILO programmes and projects, ILO, 1995 and the Guidelines to promote complementarity between standard-setting and technical cooperation, ILO, 1997 (forthcoming).
Updated by PH. Approved by EB. Last update: 15 November 1999.
