Tools: Evaluation of quality apprenticeship programmes

A number of tools for carrying out tracer studies, cost–benefit analysis, assessment of the apprenticeship policy environment and self-assessment of apprenticeships, as well as performance indicators, are given below for both policy-makers and practitioners who are engaged in the evaluation of the apprenticeship system and programmes.

Tools for conducting tracer studies

Online survey for graduate apprentices, EAN

This tool is an example of an online survey for graduate apprentices, to assess their opinions about the quality of apprenticeship programmes. The #AskTheApprentices survey questions cover modules such as: the quality of education, learning materials and teaching methods; apprentices’ rights, responsibilities and protection; apprentices’ contracts; representation in the workplace; promotion of apprenticeships; anti-discrimination measures; and access to information on apprenticeships.

Source: https://www.surveygizmo.eu/s3/90161249/EAN-Survey.

Guide to tracer studies, ETF, ILO and Cedefop

This guide offers a step-by-step introduction to the wide range of different tracer study approaches available and provides detailed instructions on how to design a graduate tracer study, develop the questionnaire and carry out the data analysis. Furthermore, it provides two versions of the questionnaire: a minimal version, which was designed to be used with very few adaptations/changes; and a module version, which is a toolbox of different topics and questions from which elements can be selected as appropriate.

Although this guide primarily targets TVET and higher education, it can easily be adapted to apprenticeship programmes.

Source: www.ilo.org/skills/areas/skills-training-for-poverty-reduction/WCMS_534331/lang--en/index.htm.

A survey of apprentices to assess the system, United Kingdom

This report presents the findings of the apprenticeships evaluation learner survey 2017 in the United Kingdom. The survey, which is carried out on a regular basis, is intended to monitor key progress indicators and assess the impact of recent policy reforms in order to help shape future development of the apprenticeship programme. Specifically, it covers individuals’ motivations for undertaking their apprenticeship, their experience of the training they received, their satisfaction with the apprenticeship and the impact it has had on their careers.

Source: https://assets.publishing.service.gov.uk/government/uploads/ system/uploads/attachment_data/file/659709/Apprenticeships_evaluation_2017-learners.pdf.

Tools for enterprises, including cost–benefit analysis

QualiCarte – a tool that allows employers to self-assess how well they manage apprentices, Switzerland

QualiCarte is an instrument for assessing the quality of in-company vocational training in any occupation. It consists of 28 quality requirements, divided into five chapters, covering: monitoring of quality improvement measures and objectives, commitment, introduction to training, training, responsibility of the host company and end of the agreement.

This tool can be used by employers for self-assessment purposes but also functions as an external assessment tool in cases where representatives of cantons or professional associations want to assess the quality of training provided in a given host company.        

Source: http://vpet.ch/dyn/bin/21423-23923-1-qualicarte_eg_2016i.pdf.

A review of methodologies for measuring the costs and benefits of in-company apprenticeship training, ILO

Understanding the costs and benefits of apprenticeship training, as well as its outcomes, is an important factor for enterprises, enabling them to make informed decisions relating to the skills development of current and future employees. Existing studies on this topic paint a mixed picture: some reveal net benefits to employers and others, net costs. In addition to the observed variations in apprenticeship systems and practices between countries, a proper international comparison of cost–benefit analyses is hampered by differences in research methods. This report reviews the research methods and findings of major existing studies with the aim of helping to advance discussions on research methods and highlighting areas where knowledge gaps exist.

Source: http://ilo.org/skills/areas/work-based-learning/WCMS_725494/lang--en/index.htm.

Assessment of returns on apprenticeship investment, India

This tool provides a questionnaire for assessing returns on investments in apprenticeships.

The publication discusses five case studies in which senior managers from SMEs assess the costs and benefits of the apprenticeships they provide. The case studies provide a clear insight into how the cost–benefit or returns on investment (ROI) approach can be adapted to the case of SMEs in the Indian context and how evidence gained from the ROI approach can be used as an effective instrument for enterprises to base their training and hiring decisions on.

Source: https://www.ilo.org/wcmsp5/groups/public/---asia/---ro-bangkok/---sro-new_delhi/documents/publication/wcms_332263.pdf.

Analysis of the costs and benefits of apprenticeship to employers, Germany

This tool provides information on how dual vocational education and training is financed in Germany and which cost–benefit factors are involved.

The tool answers the following questions:

  • What are the actual costs and benefits of vocational education and training in Germany?
  • Is external recruitment cheaper than training?
  • Is dual VET a worthwhile model?
Source: https://www.bibb.de/govet/en/71186.php.

Performance indicators

Set of indicators for assessing work-based learning, IAG

This paper lists a range of indicators developed by the Inter-Agency Working Group (IAG) on Work-based Learning (WBL) to monitor and evaluate WBL. Although the proposed indicators are related to the performance of WBL, they can easily be adapted for the assessment of quality apprenticeships.

The proposed indicators cover two policy areas:

  • Access, participation and equity – the indicators show the proportion of learners that participate in WBL, and the proportion of companies that provide WBL.
  • Quality and efficiency of WBL – the indicators show the proportion of learners that complete WBL programmes and the proportion of companies with a certified in-company trainer/mentor.

There are also two additional indicators for quality assessment that can be used during a transition period when WBL programmes are not yet fully developed: share of programmes with learning outcomes/objectives for the WBL component and share of programmes with assessment procedures for the WBL component.

Source: https://www.skillsforemployment.org/KSP/en/Details/?dn=EDMSP1_254639.

Apprenticeship outcomes performance matrix, United States

In the United States, registered apprenticeship programmes are evaluated using an apprenticeship outcomes performance matrix, which lists indicators for employers, apprentices or workers, and partner organizations.
Indicator Outcomes Time frame
Employers
Number of apprentices retained    
Decrease in recruitment costs    
Increase in productivity    
Increase in workforce diversity    
Increase in workplace safety    
Apprentices/workers
Wage increases    
Number of credentials earned    
Partner organizations
Number of programmes developed  
Number of individuals trained    
Increase in under-represented populations engaged (e.g. women in non-traditional occupations, veterans, individuals with disabilities, etc.)    
Number of apprentices retained    
Source: United States Department of Labor, n.d.

Tools for system-level evaluation

The ILO has developed an analytical framework for evaluating and benchmarking a country’s apprenticeship system and programmes. This framework can  assist in carrying out detailed, external evaluation of the policy environment and apprenticeship system, leading to practical policy recommendations to overcome identified challenges. It enables a rapid assessment of an apprenticeship system to be carried out and highlights gaps and challenges that are not always obvious to the practitioners themselves.

Evaluation tool for apprenticeship policy and system, ILO

The tool serves three purposes, as illustrated below:

  1. Comprehensive evaluation tool

    The tool, in the form of an analytic framework, provides guidance on carrying out an evaluation of a country’s apprenticeship system, offering an analysis of the characteristics and performance of the apprenticeship law, policy and system in a country. It also allows the country’s system to be benchmarked against international good practices.

    It describes the evaluation procedures and includes three questionnaires covering:

    • the public authorities responsible for apprenticeships
    • the social partners
    • apprenticeship graduates.
  2. Rapid self-assessment tool

    The tool can also be used by practitioners to carry out a rapid self-assessment to help identify the perceived strengths and challenges in an apprenticeship system. As the self-assessment can be completed quickly, and by a range of different stakeholders, it might be used as part of a workshop involving apprenticeship stakeholders. Completing the self-assessment would form the initial point for debate and discussion, allowing individuals to explore those areas where different stakeholders’ assessments of the system are radically different, determining why those differences of perception arise and what steps are necessary to improve system performance.

  3. Tool for understanding policy environment

    The tool can also be useful in helping practitioners to develop a clear understanding of the policy environment, including national law, policies and regulatory and institutional framework, which is necessary before embarking on the development of apprenticeship programmes.

Source: ILO, 2020a.

The following two tools are in the form of reports that provide the results of reviews of a country’s apprenticeship system, carried out by CEDEFOP and OECD. These reports illustrate examples of actual reviews, using methodology and tools developed by the two organizations.

Review of the apprenticeship system in Italy, Cedefop

Cedefop has been carrying out in-depth reviews of apprenticeship systems at national level in order to identify their specific strengths and any challenges that they face in order to propose a set of policy recommendations for ensuring quality apprenticeshpis. Such information can also help other countries to reflect on their own practices and implement reforms for quality apprenticeships.

This report includes the key findings, conclusions and recommendations of the thematic country review on apprenticeships in Italy. It also offers a short description of the review’s rationale and methodology.

Source: https://www.cedefop.europa.eu/files/4159_en.pdf.

Evaluation framework for Modern Apprenticeships in Scotland, OECD

This evaluation framework aims to generate insights into the outcomes, benefits and challenges of Modern Apprenticeships in Scotland, thereby strengthening the links between apprenticeships and labour market policies. Furthermore, it can also provide policy-makers in other countries with a concrete example of an evaluation strategy in an actual policy setting.

The evaluation framework proposes recommendations for evaluating Modern Apprenticeships in Scotland, detailing various evaluation activities, scope of evaluation and data options. It outlines both long-term and medium-term evaluation strategies covering various methodologies, such as ex-ante strategy, use of multiple specific control groups, analysis of impact on well-being and on employer productivity, etc.

Source: https://www.oecd-ilibrary.org/docserver/59084781-en.pdf?expires=1568268999&id=id&accname=guest&checksum=730E7BACD41ADCCF809CD54135EE1894.