Deadline: 31.12.2024, 17:00

This document provides the Terms of Reference (ToR) for an external Final Evaluation of the Project “Aligning Education and Training with Labour Market Needs – ALLED Phase II” (in further text ALLED2)1. A mid-term evaluation of the ALLED2 was conducted in 20212 with an ensuing Management Response.

This ToR provide background information about ALLED2, the purpose of the evaluation, its objectives and scope, the evaluation questions, the envisaged methodology as well as qualifications and selection criteria for the evaluator(s).

The maximum available budget for this evaluation amounts to EUR 80.000, the evaluation is planned to be conducted from January to July 2025.

The Austrian Development Agency (ADA) invites eligible and qualified experts to submit their non-binding proposals/applications in English, clearly indicating “Final Evaluation ALLED2” in the subject line. Proposals must be submitted via email to [email protected] by 31 December 2024, 17:00. (CET).

Clarification Inquiries: Questions regarding the proposal process can be directed to [email protected] prior to 27.12.2024 at 17:00 (CET). Terms of Reference (ToR)

Final Evaluation of the project

ALLED2 Aligning Education

and Training with Labour

Market Needs

Implementation Country: Kosovo

EU Contract Number: IPA/2018/402-911

ADA Project Reference Number: 6531-00/2018 and 6531-01/2018

Project Timeframe: 1 March 2019 to 30 April 2025

Name of Partner Organisation: Austrian Development Agency (ADA)

1. Context

This document provides the Terms of Reference (ToR) for an external Final Evaluation of the Project “Aligning Education and Training with Labour Market Needs – ALLED Phase II” (in further text ALLED2). A mid-term evaluation of the ALLED2 was conducted in 2021 with an ensuing Management Response. 

This ToR provide background information about ALLED2, the purpose of the evaluation, its objectives and scope, the evaluation questions, the envisaged methodology as well as qualifications and selection criteria for the evaluator(s).

The maximum available budget for this evaluation amounts to EUR 80.000, the evaluation is planned to be conducted from January to July 2025.

2. Background

The project “Aligning Education and Training with Labour Market Needs – ALLED Phase II” (in further text ALLED2)” is funded by the EU, as a part of IPAII (2017) for Kosovo with co-funding by the Austrian Development Cooperation. According to the IPA 2017 financing decision project is implemented by Austrian Development Agency (ADA) via Indirect Management (IM). The total budget of the project amounts to EUR 5.000.000, whereas the funding from the EU is EUR 3.800.000 and ADA’s contribution accounts for EUR 1.200.000.

All planned actions and activities aim to support Kosova’s Ministry of Education, Science, Technology and Innovation (MESTI) and Employment Agency of the Republic of Kosova (EARK)/ Ministry responsible for Labour and Social Welfare (in the original project documentation MLSW) in reforming the education and training system to link education and training with labour market needs, and to reflect the European Framework of Key Competencies. 

The specific aim of ALLED2 is to strengthen the quality and relevance of education and training programs in relation to labour market demands while adapting the legislative framework and mechanisms necessary for increased employability. The project emphasizes building concrete cooperation models between education and the private sector, aiming for systemic effects that ensure a more integrated approach to education.

By adopting an innovative, structured methodology based on the triple helix principle—which promotes collaboration between the public sector, academia, and business—the ALLED2 project seeks to create a responsive Vocational Education and Training (VET) system that contributes to economic growth and overall employability.

The project focus is on upgrading VET programs and enhancing the quality of teaching and learning processes in key sectors in Kosovo, including agriculture, food processing, mechanical engineering, and energy. These sectors were chosen based on the findings and recommendations from the Labour Market Needs Analysis – Perspectives for the Future in Kosovo (July 2019, ALLED2).

Building on the successes of the previous ALLED1 project, ALLED2 aims to incorporate principles of environmental protection, gender mainstreaming, and social inclusion throughout all project activities and results.

The intended results of ALLED2 as per project document are:

ER 1: Study programmes in public Higher Education (HE) institutions serve as a competence hub for the industry and VET.

ER 2: Professional qualifications and occupational standards meet NQA criteria for validation.

ER 3: VET schools and VTCs offer qualitative programmes based on the Labour Market Needs.

ER 4: Education and training system-related governance procedures support the alignment of education and training with labour market needs.

ER 5: The link between the VET and the business sector is more structured.

The project’s direct beneficiaries include:

• Ministry of Education, Science, Technology and Innovation (MESTI)
• Employment Agency of the Republic of Kosovo (EARK)
• Public universities
• Municipal Education Departments (MED)
• Vocational Education and Training (VET) schools
• Vocational Training Centres
• Kosovo’s Accreditation Agency (KAA)
• National Qualifications Authority (NQA)

The project’s indirect beneficiaries encompass:

• The Higher Education sector
• VET and Vocational Training Centre (VTC) students
• Job seekers
• The private sector
• Adult learners

Project Governance: The donors, along with representatives from various government agencies and beneficiary institutions, form the project’s governance structure through the Steering Committee (SC) and the Advisory Committee (AC). The SC includes high-level representatives from key institutions, with one representative each from the Ministry of Education, Science, Technology and Innovation (MESTI), the Ministry of Labor and Social Welfare (MLSW), the European Union Office in Kosovo (EUOK), and the Austrian Development Agency (ADA) as voting members. ADA, as the lead implementing agency, has the final decision-making authority in cases of tied votes. The committee also includes the European Training Foundation (ETF) as a non-voting observer and the Team Leader as a non-voting member. The SC can invite representatives from other institutions or individual experts to participate in meetings on specific topics, though without voting rights. Additionally, the AC is composed of professionals from the beneficiary institutions, enhancing project alignment and stakeholder collaboration. 

3. Purpose and Objectives of the Evaluation

The Final Evaluation is commissioned by the Austrian Development Agency (ADA) at the conclusion of the ALLED2 Project implementation, in line with the project’s Description of Action (DoA) of the ALLED2 project. This external, independent evaluation aims to ensure accountability and foster learning within the context of the project. It is anticipated to commence in January 2025 and be concluded in July 2025.

The evaluation will assess the project’s effectiveness, impact, and sustainability, aiming to promote a culture of continuous improvement and to provide evidence that informs future programming and implementation. The primary users of the evaluation results will be the donors and ADA staff at both headquarters and field offices.

4. Scope

The scope of the Final Evaluation will cover the entire project duration, from March 1, 2019, to April 30, 2025, encompassing all components of the project. The following OECD-DAC evaluation criteria will be within the scope of this evaluation: relevance, effectiveness, and sustainability.

The evaluation will include field visits to beneficiary institutions across 17 targeted municipalities in Kosovo (including VET schools and Vocational Training Centers), with additional engagement with national stakeholders. The specific locations for data collection in the field will be identified and selected based on a range of criteria during the inception phase. 

5. Evaluation Questions

Relevance

   1.  To what extent do the project’s objectives align with Kosovo’s current government institutional needs, and the broader goals of relevant national and local organizations priorities in the area of VET?

  2.  How has the project responded to the changing needs of its primary beneficiaries, including students, VET institutions, and private sector partners? What have been hindering and facilitating factors for continuing relevance in this context?

Effectiveness

  3.  To what extent has ALLED2 managed to support an effective cooperation between VET Schools and businesses? What have been hindering and facilitating factors?

   4.  How has the project’s data collection and quality assurance mechanisms contributed to improvements in evidence-based policymaking, VET teacher qualifications, and educational standards in Kosovo? 

  5.  What have been hindering and facilitating factors in relation to results achievement and how have project management and implementing partners addressed these? 

Sustainability

   6.  Which tools, practices, or institutional mechanisms if any, established within the framework of the project are likely to be sustained after the end of the project and how? 

    7.  What institutional capacities have been embedded in the system, and what potential do the partnerships, collaborative frameworks, and structures established by the project hold for creating lasting impacts in the VET sector? What evidence of long-term commitment or lack thereof is evident among key stakeholders? 

6. Design and Methodological Approach

The evaluation must be conducted in a gender-sensitive and inclusive manner, respecting the principle of “do no harm” throughout the process. To ensure comprehensive insights, a mixed-methods approach will be employed, drawing from both qualitative and quantitative methods. This approach strengthens internal validity by using data and method triangulation, improving the reliability of the evaluation outcomes.

A broad range of data sources and collection methods will be employed to ensure reliability, minimize bias, and promote impartiality, ensuring that findings are rooted in relevant and robust information. Likely methods include:

• Document review of relevant project materials, such as monitoring data, the mid-term evaluation, and the EU’s Results-Oriented Monitoring (ROM) and statistical data.
• Key informant interviews with project stakeholders.
• Focus group discussions with beneficiaries and other key groups.
• Conduct of a survey document to support quantitative insights.
• Direct observations for on-the-ground assessment.

During document review, available monitoring and evaluative data will be essential to creating a solid evidence base for further analysis.

7. Process, Deliverables and Timetable

7.1 Process

The evaluation will unfold in three phases: inception, inquiry, and synthesis, each with specific milestones and deliverables:

1. Inception Phase: Detailed evaluation planning, including method refinement and selection of data sources.
2. Inquiry Phase: Data collection and analysis.
3. Synthesis Phase: Final analysis, reporting, and validation of findings.

Formal feedback loops will be applied to review and finalize draft evaluation products, such as the inception and evaluation reports, using structured feedback matrices to ensure clarity and efficiency. A minimum of ten working days is required for the Austrian Development Agency (ADA) to review and provide feedback. Only fully completed drafts, inclusive of necessary annexes, will be submitted for formal feedback.

For required structure and content of both the Inception Report (IR) and Evaluation Report (ER), please consult Annex 5 (Quality Checklist for IR) and Annex 6 (Quality Checklist for ER) of the ADA Guidelines for Programme and Project Evaluations (2020).

Inception Phase

The inception phase begins with a kick-off meeting and concludes with the approval of the inception report. Upon contract signing, a date and time for the kick-off meeting will be arranged between the successful bidder and ADA (HQ, project team, and the Coordination Office in Prishtina). In preparation, ADA will provide the evaluation team with key documents, including project documentation and a draft stakeholder list.

The kick-off meeting will serve as a critical introduction to the project’s background, allowing evaluators to understand its context, evolution, and current status. It will also provide an opportunity to discuss data availability, logistics, and potential challenges or limitations. Additionally, the meeting will be essential for clarifying expectations and setting a timeline for subsequent steps.

The Inception Report (IR) is the first key deliverable for the evaluators. In this report, evaluators must present the detailed evaluation design, outlining the approach to answering each evaluation question and specifying how information will be gathered and analysed. Exploratory interviews may be conducted during this phase to inform the evaluation design. The work plan outlined in the IR should be feasible and aligned with the evaluation objectives. For guidance on structure and content, evaluators should consult the Quality Checklist for Inception Reports in ADA’s Guidelines for Programme and Project Evaluations (Annex 5). This phase concludes upon written approval of the inception report by ADA.

Inquiry Phase

The inquiry phase encompasses both data collection and data analysis. Following the approval of the inception report, data collection will commence through desk research and fieldwork in Kosovo and Vienna, conducted both in-person and virtually. Once the data collection in Kosovo is complete, a debriefing with ADA will be scheduled to update the project team on progress, highlight any encountered challenges or data gaps, and agree on follow-up steps if necessary. Additional data collection or key informant interviews may be organized as needed.

After data collection, and approximately two weeks into data analysis, preliminary findings will be presented to ADA (HQ, project team, and Coordination Office in Prishtina) in a virtual meeting. This interim discussion will allow ADA to review initial results, provide feedback, and guide further analysis.

Synthesis Phase

The synthesis phase involves drafting, reviewing, and finalizing the evaluation report. ADA’s Guidelines for Programme and Project Evaluations (Annex 6) should be referenced for the structure and content requirements of the evaluation report. Feedback from ADA will be integrated into the draft reports through structured reviews, with final approval marking the completion of the evaluation.

The synthesis phase concludes with ADA’s written approval of the final evaluation report.

7.2 Deliverables

The consultants are expected to provide the following deliverables:

• Inception Report (for structure and content requirement see Annex 5, ADA PP Evaluation Guidelines, pp. 46-47): A draft and final inception report, 10-15 pages in length, plus annexes.
• Evaluation Report (for structure and content requirement see Annex 6, ADA PP Evaluation Guidelines, pp. 48-49): A comprehensive evaluation report, including both a draft and a final version, with a main body of 25-30 pages (excluding annexes and the ADA Results Assessment Form [RAF]). 
• Infographic Summary: A one-page infographic summarizing the evaluation facts and results in English, to be submitted alongside the final report.

All deliverables must be in English. The Executive Summary of the Evaluation Report, along with the one-page infographic, must also be in English.

The quality and acceptance of both the inception and evaluation reports will be based on compliance with the quality checklists provided in Annex 5 and Annex 6 of ADA’s Guideline for Programme and Project Evaluations (July 2020).

7.3 Tentative Timetable

The total estimated working days are around 130. The Final Report is due by 30 June 2025.

 

8. Evaluation Team

The final evaluation team should consist of four multidisciplinary members (both national and international), with one designated as the team leader. The team must be gender-diverse and possess the following qualifications:

• Academic Background: Each team member must hold a relevant academic degree
• Evaluation Experience: All team members must have a minimum of three years of experience in designing and conducting evaluations for complex programs and projects, supported by three reference evaluation reports.  
• Specialized Expertise:
• Education Sector: At least one additional team member must have proven expertise in the educational sector, including VET.
• Evaluation: the team lead must have proven experience in evaluating complex projects/programs in the educational sector, including VET. 
• Project Cycle Management: All team members must be experienced in project design, Theory of Change, and project monitoring.
• EU and Multi-Donor Projects: At least one team member must be familiar with EU-funded or multi-donor projects.
• Cross-Cutting Issues: The team must have proven expertise in gender equality, environmental sustainability, and social inclusion.
• Social Research Methods: The team should be skilled in social research, including participatory methods.
• Language and Communication Skills:
• Excellent written and spoken English skills are required for all team members.
• The evaluation team must be able to conduct data collection in local language 
• Strong presentation skills for sharing findings and insights effectively.

Note: To ensure objectivity, team members must not have been involved in any aspect of the design, implementation, or monitoring of the ALLED2 project.

  9. Evaluation Locations, Financial, and Logistical Arrangements

   9.1 Evaluation Management Arrangements

The evaluation management will be primarily conducted in Pristina, with fieldwork planned across Kosovo. Field visits will cover various locations across Kosovo, with each stakeholder represented by at least one tailored evaluation instrument. Specific sites for these visits will be determined in consultation with the ALLED2 PIU. Evaluators are responsible for organizing field visits independently, including scheduling appointments with relevant stakeholders, the project team will support evaluators in identifying relevant contacts within these organizations as necessary.

The ADA Coordination Office in Pristina and ADA Headquarters in Vienna will also facilitate communication and information exchange. 

All evaluation activities will adhere to ethical standards and guiding principles, ensuring impartiality and independence throughout the process.

9.2 Budget and Payments

The Contractor will receive a fixed remuneration, along with reimbursement for documented expenses.

Payments will be made in three instalments, contingent upon the completion, feedback, and acceptance of key deliverables by ALLED2. The final instalment will be disbursed upon delivery of the approved Final Evaluation Report. Payments will be processed within 10 working days following the submission, ADA’s and ALLED2’s satisfactory review, and formal approval of the services as outlined in the Terms of Reference. 

In addition to the fixed remuneration, expenses (e.g., business trips) will be reimbursed in alignment with the budget line “2. Travel and Accommodation Expenses” as specified in the Annex 2 Budget template. All expenses must be supported by a detailed financial statement and original vouchers submitted to ADA.

Payment Breakdown:

• Inception Report: Based on actual expenses, up to a maximum of 30% of the contract sum.
• Draft Evaluation Report (including draft executive summary and Results-Assessment Form): Based on actual expenses, up to a maximum of 40% of the contract sum.
• Final Evaluation Report (including final executive summary and Results-Assessment Form): Based on actual expenses, up to a maximum of 30% of the contract sum.

The final payment will be made upon the submission and approval of the Final Evaluation Report.

  10. Coordination and Responsibility

Support Provided by ALLED2

ALLED2 will maintain regular communication with the contractors, offering feedback, guidance, and necessary support to help achieve the project objectives. This ongoing engagement will ensure that any potential issues related to performance and work quality are promptly addressed. Contractors will receive essential information about the project, including relevant reports and documentation of previous marketing activities.

For this evaluation, the primary contacts will be:

• Mr. Adnan Ahmeti (ADA Coordination Office)
  Email: [email protected]
  
• Ms. Blendina Kuci (ADA Coordination Office)
  Email: [email protected]

  11. Technical and Financial Offer

The applicant is required to submit a technical and financial offer, including the following components:

Technical Offer (maximum 8 pages), including:

• Understanding of the Assignment: Overview of the applicant’s interpretation of the assignment’s objectives and requirements.
• Evaluation Approach and Methods: Description of the suggested evaluation methodology, including measures for stakeholder and beneficiary engagement, as well as health and safety considerations.
• Evaluation Plan, covering:
• Team Structure: Outline of team roles and responsibilities.
• Operational Work Plan: Detailed plan with estimated working days per task and team member.
• Timeframe: Proposed timeline for the evaluation.
• Evaluation Team CVs: Curriculum vitae for each member of the evaluation team.
• Capacity and Past Performance: Institutional and/or individual qualifications relevant to the assignment, demonstrating experience in similar projects.
• Evaluation Report Samples: At least two examples of previous evaluation reports authored by the team.
• Independent References: A minimum of two signed references from separate, independent sources.

Financial Offer

• Budget: A comprehensive budget in EUR (excluding VAT), detailing costs associated with each deliverable in the form of the Budget Template (Annex 2).
• Company Identification: Organization Identification Number (if applying as a company).

For an example of an evaluation infographic, refer to: Evaluation Infographic Example. Requirements for the structure and content of the Executive Summary can be found in Annex 6 of the ADA Guidelines on Programme and Project Evaluation (Section 1, page 48).

ADA will score bids based on both the technical (80%) and financial (20%) components of the offer.

Financial Proposal Requirements:

• Fee and Expense Breakdown: Include a detailed breakdown of fees and itemized reimbursable expenses. Specify separate pricing for in-person versus remote working days.
• Travel Expenses: Provide a distinct breakdown for travel-related costs.
• Budget Template: A budget template is provided below and included in Annex 2 of the Terms of Reference (ToR).
• Compliance: The financial proposal must strictly adhere to the provided budget template (Annex 2). Non-compliance, such as using an alternative format, will lead to proposal rejection.

All technical evaluation team members are required to use their own laptops and submit deliverables digitally. The offeror is responsible for planning the team’s workspace, conference facilities, telecommunication, printing, and other essential resources to fulfill the contract activities.

  12. Evaluation Criteria and Rating

The selection will be based on the following criteria and scoring:

In the final scoring, a comprehensive approach will ensure that each proposal is evaluated based on the alignment of its technical and financial offers with ADA’s requirements and goals for this evaluation.

  13. Submission of Proposal/Application

The Austrian Development Agency (ADA) is conducting a direct contracting process to hire the evaluation team. Several experts and companies will be invited to submit proposals. Additionally, a call for expressions of interest, including the Terms of Reference, will be published across various platforms and networks.

Data Privacy: Personal data included in the proposals (such as CVs) will be stored and utilized internally by ADA during the proposal review process. It is important to highlight that consultants must not have been involved in the design, implementation, or monitoring of ADA’s engagement in Kosovo during the evaluation period.

Submission Guidelines: 

The Austrian Development Agency (ADA) invites eligible and qualified experts to submit their non-binding proposals/applications in English, clearly indicating “Final Evaluation ALLED2” in the subject line. Proposals must be submitted via email to [email protected] by 31.12.2024, 17:00 (CET).

Please note that any offers submitted in a different format or sent to an alternative email address will be excluded from the evaluation process.

Submissions should consist of two separates, signed documents in PDF format:

• One clearly marked TECHNICAL PROPOSAL
• One clearly marked FINANCIAL PROPOSAL, utilizing the Budget template provided in Annex 2.

Please include “Final Evaluation ALLED2” in the subject line of your email. All offers must be signed and dated.

Restrictions on Submission: Proposals submitted via cloud-based services (e.g., WeTransfer) will not be considered eligible for evaluation. Only electronically submitted applications will be accepted.

Clarification Inquiries: Questions regarding the proposal process can be directed to [email protected] prior to 27.12.2024 at 17:00 (CET).

Rights Reserved: The ALLED2 project/ADA, reserves the right to reject any and all proposals and to cancel, reschedule, or terminate the call for expressions of interest and/or tender at any point in time.

  14. Other Conditions

Processing of Personal Data:

During the initiation and execution of contracts, ADA may process personal data related to individuals collected from prospective contractors or disclosed by third parties under their direction. This includes personal data of employees, legal representatives, agents, or other partners of the prospective contractors.

  14.1 Confidentiality

All materials produced during the consultancy period will be treated as strictly confidential. The rights for distribution and publication will solely reside with ALLED2/ADA. The contractor is prohibited from using or republishing any materials without explicit permission.

  14.2 Penalty Clause

The contract will include a penalty clause stipulating that ALLED2/ADA reserves the right to withhold payment or reduce the payable amount if one or more requirements established for this assignment are not met, or if deadlines for task completion are missed.

  14.3 Contract Duration

The selected company will be awarded a contract that commences on the date of signature and concludes in July 2025.

  14.4 Copyrights & Utilization Rights

All materials produced during the assignment, whether raw or edited, will be the intellectual property of ALLED2/ADA.

  14.5 Code of Conduct

All consultants must adhere to the ALLED2/ADA Code of Conduct throughout the consultancy period. 

Annexes

• Guidelines for Programme and Project (PP) Evaluations at the Austrian Development Agency (ADA), can be downloaded from the ADA website: https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf
• The RAF template (Excel format) can be downloaded from the ADA website: www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierung_Templates/Annex9_Results_AssessmentForm_Template.xlsx