Project Evaluation

Terms of Reference

Setting High-Quality Preschool Standards Reinforcing Education, Science and Economy (SHPRESE)

General information

Job title:                                                          International and local evaluation consultants for the evaluation of the SHPRESE project

Project title:                                                   Setting High-Quality Preschool Standards Reinforcing Education, Science and Economy (SHPRESE)

Donors:                                                           The Government of the Principality of Liechtenstein, Medicor Foundation and Swiss Agency for Development and Cooperation (SDC)

Project country:                                           Republic of Kosovo

About Caritas Switzerland in Kosovo

Caritas Switzerland (CaCH) is an independent Swiss NGO that is engaged in Switzerland and in more than 20 countries worldwide, including Kosovo. Our mission is to prevent, combat and eradicate poverty in a sustainable manner. To achieve this goal, Caritas employs a policy of equity: we support inclusive communities with opportunities for everyone regardless of their ethnic background, group affiliation or gender. Caritas Switzerland has been working in Kosovo since 1999 beginning with humanitarian interventions after the war but shifting soon after into development cooperation. Currently, CaCH is engaged in further improving the quality of pre-primary education (PPE) and the services offered in this level of education through its SHPRESE project, all the while supporting and strengthening the cooperation with local institutions. In this context, the enhancement of pre-primary education quality goes hand in hand with the improved and sustainable government service in training and capacity building of in-service PPE educators, capacity building of pre-service educators through the work conducted jointly with the Faculties of Education (FEs),as well as the promotion of PPE educators and their professional development at the municipal level. Our approaches put emphasis on effective teaching-learning strategies such as learning-by-playing approach and have  narrowed the focus on the grade 0 / preparatory grade for children aged 5 to 6. Moreover, we work with the universities to ensure long-term improvement of quality in PPE and employ a participatory and culturally sensitive approach with relevant public institutions. Apart from the national level, we also work at the municipal level by involving and building the capacities of relevant staff of education departments.

Project Context and Background

Field of activity:	Education
Project Duration:	01.07.2018 – 30.06.2022
Project area:	Republic of Kosovo – nation-wide
Implementing partners:	Caritas Switzerland in Kosovo (CaCH) – lead organization Kosovo Pedagogical Institute (KPI) Association of Kosovo Municipalities (AKM)
Other partners:	Ministry of Education, Science, Technology, and Innovation 4 public Universities / Faculties of Education in Gjakova, Gjilan, Mitrovica, and Prizren 20 Municipal Education Directorates
Target groups:	Children of age5 to 6 years; current and future preschool educators; institutions on the national, local, and academic levels
Funded by:	The Government of the Principality of Liechtenstein, Medicor Foundation, Swiss Agency for Development and Cooperation (SDC), Caritas Switzerland with  co-funding from partner municipalities

Early childhood education, including the pre-primary level, continued to remain neglected segments in the education sector. This level has been accompanied with low quality in terms of teaching approaches and classroom environments. In 2018, when the SHPRESE project began, PPE continued to remain not inclusive and mandatory for all children with only around 85% attending this level of education. Moreover, while the quality training and practical approaches for educators are limited, a system for monitoring the quality of education was also inexistent. To contribute to the improvement of quality education and service provision in pre-primary education in Kosovo, as of July 2018, Caritas Switzerland (CaCH) implements the SHPRESE project, which will end in June 2022.

To achieve its overall objective, the project includes three main components:

• The improved quality of PPE in Kosovo through improved and sustainable government service in training and capacity building for existing pre-primary educators at municipal level and by creating a child-friendly environment in pre-primary classrooms;
• The enhanced capacity of future PPE educators (students) in contemporary methodology in PPE;
• Sustainable coordination among relevant stakeholders in PPE at national level established and led by the Ministry of Education.

Purpose and Objectives of the Project Evaluation:

The purpose of this project evaluation is to provide an independent assessment that focuses on: Lessons learned from the pervious and actual interventions. As well as conclusions and recommendations including assessing needs, potentials, and the scope for continuation. The evaluation should be conducted through the OECD DAC criteria which includes an analysis of relevance, coherence, effectiveness, efficiency, impact, and sustainability. The specific objectives of the evaluation are the following:

1. Assess the implementation of the project to date, identifying successful elements and areas for potential advancement;
2. Analyse the implementation strategies of the project with regard to their potential effectiveness in achieving the project outcomes and impacts; including expected results as defined in the logframe;
3. Assess the role of the project team towards project implementation together with strategic partners and other relevant stakeholders;
4. Identify lessons and potential good practices for key stakeholders in the pre-primary education level;
5. Provide strategic recommendations for different key stakeholders for future interventions.

Scope of the evaluation

The project evaluation will cover the whole implementation period until the time of evaluation, this is from 1^st July 2018 until present. The evaluation will analyse the whole scope of the project and its logical framework which delineates the overall objective, specific objectives, results, and activities.

Evaluation Questions

The 6 DAC criteria will be taken into consideration namely: relevance, coherence, effectiveness, efficiency, impact and sustainability. The external evaluation team is asked to pay particular attention to the following issues:

• Relevance and Coherence of the Project: What interventions in pre-primary education were intended and have those changes been achieved:
• To what extent was the project relevant to the identified needs of the beneficiaries and existing gaps in the pre-primary education level?
• To what extent have the interventions responded to partner/institution needs, in particular the governmental structures like public schools, education directorates, Ministry of Education and others?
• To what extent are the strategies and interventions of the project in line with the overall pre-primary education development context in Kosovo (incl. strategies, policies, curriculum, etc.)?
• To what extent is the project in line with the current trends and developments in the field of pre-primary education?
• To what extent have the interventions complemented other initiatives in the country?
• Effectiveness of the Project: Evaluation of the overall project implementation compared to the plans and targets: Is the project on-track to achieve the expected targets by completion:
• To what extent did the project achieve its overall and specific objectives?
• How effective were the strategies and the tools used in achieving the projects objectives?
• To what extent and if, how was the “learning through playing” approach embraced by the beneficiaries and partners/institutions?
• To what extent does the project reach its target group and how far have project interventions contributed towards addressing their needs and constraints?
• How has the pandemic affected the project’s activities, and how has the project responded to it?
• Efficiency of implementation: Evaluation of project set-up and steering:
• Have the financial and human resources within the SHPRESE project been efficiently allocated?
• Has the implementation of the project enabled learning, knowledge exchange and synergies between the staff members and other institutions/partners?
• Is there a monitoring system in place and to what extent does the monitoring system utilised by SHPRESE ensure effective, efficient, and adaptive project management?
• How well were mandates of different partners/institutions in pre-primary education managed?
• Were the inputs realistic, appropriate, and adequate to the needs of the beneficiaries?
• Did project activities overlap with other interventions (funded by national or international donor organizations?)
• Impact: What impacts has the project achieved:
• To what extent has the project generated significant positive impact in early childhood education?
• To what extent have the practices within the institutions/beneficiaries changed / been adapted by the governmental institutions and other partners?
• Which other elements of the project have an impact beyond project outcomes foreseen?
• What effects are consequential of the interventions conducted through SHPRESE?

• Sustainability: How sustainable are the interventions implemented in the project:
• What is the likelihood of sustainability of the project outcomes after the completion of the project?
• How sustainably are the suggested practices anchored in the schools, at the universities and in the education policy institutions?
• How sustainable is the use of the classrooms installed?
• How sustainable is the teacher training anchored into the education and training system?
• How functional is the Educators Network?
• How were the capacities strengthened at the individual and organizational level?
• Are the training and equipment provided within the pre-primary education level sustainable?

Design and Approach

The evaluation consists of several phases:

• Contract and Kick-off meeting: The contract is signed, and clarification of the ToR and the assignment take place. First documents, including available data, are provided to the evaluation team as well as eventual proposed changes to ToR and approaches have to be proposed at this stage.
• Desk research: The evaluation team studies all necessary project documents and materials; re-constructs and analyses the intervention logic. Existing data needs to be analysed and interpreted.
• Briefing with the team: An introduction meeting with the project team as well as individual meetings with project management and team members take place, including evaluations of the interventions, management procedures and project set-up as well as resources.
• Inception-Phase: The inception report will serve as a roadmap for the evaluation that will ensure shared understanding between the evaluator(s) and project team concerning the workplan, deliverables and timeframes. Importantly, it will further outline the evaluation design and present the data collection and analysis methods, tools to be used and identify potential risks and limitations along with adequate mitigation strategies.
• Field visits/Interviews/Focus groups: Interviews, focus groups and field visits will be conducted with different stakeholders and partners to validate key tangible outputs and interventions.
• Debriefing/Presentation: Presentation of key findings (feedback workshop).
• Submission of the draft Final Report for revision and commenting by the project and programme team.
• Submission of the Final Report upon revision and integration of comments/suggestions.

Proposed methods:

For the different phases it is expected that data and information will be obtained through different methods such as: analysis of documents, structured and semi-structured interviews, face-to face or by phone meetings, key informant interviews, group discussions, survey, observation, and others. All data collected needs to be disaggregated by sex and ethnicity. It is expected that the evaluation team will present concrete recommendations which will be addressed to the specific stakeholders.

Evaluation Matrix

Developing the evaluation matrix is the first task and the integral part for the preparation of the inception report.  When developing the evaluation matrix, the evaluator(s) should follow the following structure, including the above-mentioned elements:

• Evaluation criteria
• Evaluation questions
• Indicators
• Sources
• Methods for data collection

The OECD DAC criteria shall be considered throughout the entire evaluation process.

Expected outputs and deliverables/Report

1. Inception report (approx. 10-15 pages) containing appropriate methodology to be applied during the evaluation, as well as the work plan to be used during the course of the assignment is drafted, submitted, and endorsed by CaCH;
2. a final draft evaluation report (about 25-30 pages without annexes), including a draft executive summary and the results-assessment form (part of the reporting requirement);
3. and the final evaluation report (25-30 pages without annexes), the final executive summary and the results-assessment form (part of the reporting requirement).

All reports need to be written in English. The executive summary should summarise key findings and recommendations (three pages maximum) and needs to be submitted as part of the final draft report.

Workplan

Up to 45 working days are currently estimated for this assignment. The final project evaluation is foreseen to start by mid of March/beginning of April 2022 and to be completed by the end of May 2022. The submission date of of the revised final report is  10. June 2022.

Activity	International Expert 1	Local Expert 1	Total days per activity
Desk Phase – Inception	4	3	7
Field Visits Phase	7	8	15
Data Analysis	5	2	7
Draft report	6	5	11
Debriefing and finalisation of the report	3	2	5
Total	25	20	45

The written feedback from CaCH, partners and the donors is planned to be given within maximum of 10 days after submission of the reports. The feedback and suggestions need to be assessed and fully integrated into the final report.

Evaluation Management Arrangements:

The evaluation team should consist of two consultants, an international evaluation specialist and a local evaluation specialist, who will work jointly to achieve the expected results. The international evaluation specialist is leading the evaluation team and is the lead author of the evaluation. The consultant will work together with the local specialist under direct supervision of the SHPRESE Project Manager. The project team will provide administrative and logistical support as needed.

The two evaluation specialists are expected to have the following skills:

• Experience in working in the international development sector
• Experience in conducting evaluations
• Experience in the education field, with a preference in early childhood
• Excellent communication skills in English
• Sound presentation skills

Additionally, the following skills are expected from:

Senior international Expert/Team Leader

General experience and skills

• At least 7 years overall professional experience and a Master’s degree in a relevant field as in Education, Social Sciences, International Development, Management and any other related university degree;
• At least 5 years of experience in working with international development and evaluating according to the OECD/DAC criteria;
• Experience in at least 3 evaluations in position as team leader for mid-term and/or final evaluation;
• Significant knowledge of learning through playing approach and other pedagogical approaches used in early childhood;
• Working experience in the Balkans is preferred; Knowledge of Albanian or Serbian is an advantage;
• Experience of programme design, monitoring and evaluation and qualitative data collection and analysis;
• Fluency in Excellent written and verbal communication skills in English.

Senior local expert

General experience and skills

• At least 7 years overall professional experience and a Master’s degree in a relevant field;
• Fluent in written and spoken English and Albanian, Serbian is considered an advantage;
• Expertise in learning through playing approach and other approaches used in early childhood;
• Expertise of local and national policies, strategies and other relevant documents and instruments for early childhood in Kosovo;
• Experience in qualitative data collection, analysis and interpretation;
• Excellent understanding of results-based evaluations and impact evaluations.

Specifications for the Submission of Offers Application process

Interested candidates/companies are requested to submit an electronic copy of their expression of interest/proposal by 6.3.2022 with the subject REF: “Evaluation SHPRESE” to [email protected] with [email protected] in CC.

Applicants must submit:

• Professional profile of the evaluating team/company (CVs of all individuals included in the consultancy team);
• The consultants’ suitability for the assignment and match with previous work experience, qualifications etc.;
• Composition and division of work between team members;
• Evaluation methodology that shall be used;
• Draft work plan and suggested timetable;
• Financial offer including budget broken down and individual daily fees;

The offer will be evaluated by using the best value for money approach (combined scoring method). Technical proposal (quality) will be evaluated on 80%, whereas the financial offer will be evaluated with 20%.

*Note: Accommodation and transportation costs for the consultants need to be included in the offer. CaCH does not cover office costs and maintenance.