WORKING
PACKAGES

In the first step of the PESHES project the goal was establishment of infrastructure for the development of system for performance based evaluation in Serbia. This goal was realized through several actions. The first one, “Analysis of the needs and priorities in Serbian society and local economy”, provided analysis and framework for dialogue with different stakeholders about actual needs and development directions in the field of value based management of Higher education institutions and system.

As the developing indicators regarding the inputs and outputs of higher education institutions in Serbia needs to address (most of) the following criteria: national higher education priorities; regional engagement, economical priorities, performed analysis showed: 1) Business entities need to have an overview of different orientations and parameters defined study programs, their market orientation and quality. In this way, it is possible to achieve a kind of feedback between higher education institutions and industry. 2) The National Employment Service and the Republic Statistical Office need to benefit from the access to real data on HE institutions and system, their organized monitoring and better connections with higher education institutions and businesses (today for example is not possible to generate information on the number of unemployed graduates originated from certain institutions). 3) Government and state institutions (policy makers) are interested to use the defined set of performance indicators to create the legal framework and recommendations for funding or financial models that should have incorporated indicators measuring quality educational process.

According to this analysis it is clear that project PESHES needs to fulfill specific actions stated in “Action Plan for Implementation of the Strategy for Development of Education in Serbia by 2020”, particularly action with code VO-ZD 09 “Introduction of quality indicators in Higher education”. This particular action has following implementation action (according to Strategy and Action plan): (1) Definition of set of indicators for monitoring of condition of HE, (2) Improvement of accreditation standards, (3) Development of model for implementation of indicators (information system). On the second place the project needs to be focused on action VO-AC05 “Introduction of ranking of study programs” with implementation actions: (1) Definition of set of indicators, (2) Analysis of different ranking programs (based on opinion of employers as well as based on knowledge of students), (3) Systematic inclusion of employers in procedure of evaluation and ranking (4) Establishment of Manual for ranking of study programs. All this implementation activities are covered by specific working packages and results of specific project activities will define these outcomes.

On the other side another set of stakeholders was included in this process through analysis of students’ and academics’ needs and priorities. In order to provide support from Universities it was important to analyze and improve capacities of Universities to monitor and measure specific performances.

Academic community and the universities in Serbia stands that developing a system for measuring performance and multi-dimensional ranking of study programs and institutions should contribute to a better, higher quality, more efficient, market-oriented and socially responsible management of study programs and universities. Evaluating of study programs and institutions should, through key performance indicators, enable focus on key processes, process benchmarking, comparison and thus improving key processes at universities in Serbia which will contribute to higher levels of education, research, development and innovation processes, higher terms of the internationalization as well as cooperation with industry locally and in the region. On the other hand all this would effect on the definition and redefine of institutional strategies.

Students expect that according to the performance-based evaluation of study programs and HE institutions, they will have opportunities to make choices that suit them best, bearing in mind the set of performance which would indicate, for them, essential parameters (for example, the number of unemployed graduates is a study program).

Analyses shows that the key performance indicators should be: 1) relevant, 2) statistically and methodologically rigorous, 3) derived from high quality, objective data sources, and where possible collected at by an independent body, as well as not easily manipulated, 4) as simple, transparent and explicable as possible, 5) have an explicit and consistently used definition, 6) collected and analyzed cost-effectively and 7) to inform and encourage policy and practice at both the national and institution level.

Purchasing of adequate equipment, licenses and software at the Serbian partner Universities has been done in order to provide improvement of Universities’ capacity for performance measurement. This activity has been performed according to current national public procurement procedures in Serbia.

During first year of project implementation while it was discussed about purchasing of equipment it had been found that there is quite different level of IT infrastructure and universities information systems development at the Universities in Serbia, so there was a need to do some changes in nature, type and specifications of equipment planed in project application. In line with

1) Reasons for project budget for equipment revision noted in EACEA Evaluation Report “The application does not specify what equipment is needed for what purpose and it is not clear why such a high number of desktops and tablets for some HEIs are needed to implement the project. The lack of clarity and detailed explanations lead to believe the Equipment budget headings is a bit over estimated and has therefore been reduced.” and

2) Results of analysis of the needs and priorities of the Serbian Universities for improvement of Universities capacity for collecting, storage, protecting and processing big data needed for smooth operation of the ICT solution for performance evaluation of study programs and institutions in Serbia (planed output for WP3) based on defined set of KPI – key performance indicators (planed output for WP 2),

the project consortium concluded that it would be the most useful for implementation of the project activities to change equipment specifications in the following way:

1) cc 65.400 euros for servers and following equipment (UPSs, RAMs and Racks) for storage and hosting KPI’s (key performance indicators) big data from Serbian Universities, hosting ICT solution for performance evaluation, hosting other developed applications for data’s import/export and for storage of published reports from big data processing,

2) cc 29.500 euros for networking equipment (routers and switches) to improve network infrastructure at the Serbian Universities for communication between universities (faculties) information systems and collecting KPI’s (key performance indicators) big data needed for performance base evaluations,

3) cc 79.500 euros for desktops, monitors and laptops for a) members of Universities’ IT teams working on the ICT solution for performance evaluation of study programs (leaded by University of Novi Sad staff) and institutions in Serbia and on the model for profiling and multi-ranking of study programs and HIE institutions in Serbia (leaded by University of Niš) and b) members of the project management team in smaller amount,

4) cc 8.700 euros for servers OS and OM license to smooth operation of the ICT solution and cc 4.100 euros for computers OS and antivirus license to smooth developing of ICT solution and to protect collected performance indicators data, and

5) cc 7.800 for other IT equipment (printers, scanners, multifunctional devices, projectors, etc.) needed for digitalization of hard copy data, printing and presenting of the project results and reports, in line to establish performance base evaluation culture in Serbian HE.

After receiving the EACEA authorization to cover with the EU contribution the costs related to the equipment listed above, University of Belgrade, as coordinator, published public call for joint procurement of equipment for all Serbian Universities.

Starting from the fact that different countries in EU have different systems it was necessary to provide analysis of performance based evaluation in EU.

Detailed analysis of existing performance measurement systems for higher education that exist in EU partner countries (Croatia, Netherlands, Italy and Spain) have been done by EU partners universities. The partners have been selected according to defined profile, University Twente as a leading institution in this filed, University of Rijeka because Croatia recently introduced performance measurement as a base for funding. The Center for Higher Education Policy Studies (CHEPS) of the University of Twente, one of lead partners in project U-Multirank (an independent ranking of world universities, prepared with support from the Erasmus+ programme), also presented U-Multirank methodology and set of performance indicators for HE institutions across five dimensions of higher education and research activities: 1) Teaching and learning, 2) Research, 3) Knowledge transfer, 4) International orientation and 5) Regional engagement, in order to establish the set of role models for development of Serbian model.

With EU partners universities’ help an overview report on performance based evaluation, performance-based funding and performance agreements in EU and World was prepared and presented, in order to analyze possible model for Serbian HE system and to establish the internationalization of possible Serbian solution and possibility of interconnection with models that are adopted in EU.

The second step “Development of system for profiling and raking of institutions and study programs in Serbia” is based on the results and analysis from previous step. After the analysis of the demands and points of views of different stakeholders in Serbia was performed as well as analysis of different systems and models at EU partner countries it is necessary to develop the system for performance evaluation, profiling and ranking of higher education institutions and Serbian educational system as well. It is clear that great diversity of potential indicators exists. On the other hand different systems have great variety in aims and great variety in features. It is important to have quality system in order to serve a greater variety in needs of students, the labor market and society.

Activity 2.1. has been conceived and formulated in the Proposal following from the bottom-up methodology, in which general aspects and rules are predefined in line to prepare the set of key performance indicators for study programs, so that the implementation of performance based evaluation of HE institutions and system as whole follows as a consequence. However, in spite of possible theoretical advantages of pursuing the bottom-up approach, the inverse top-down approach has been found to be more adequate, having in mind versatility of study programmes in Serbia and number of programmes (cc. 3.000 study programmes within different fields, in 3 study levels and 2 type of studies).

In line with results of performed analyses in WP1.1. and inputs from the Ministry of Education, Science and Technological Development on strategic plans for improvement of Serbian HE system quality and developing new model for funding HE institutions (universities, faculties and colleges of applied science) in Serbia, Project Coordination Board (PCB) and National Project Coordination Team (NPCT) concluded several times at its meetings that top priority is to develop the set of key performance indicators for performance-based evaluation of Serbian HE institutions (faculties and colleges of applied science) with plans to use part of defined indicators for performance evaluation at level of universities and study programs, as well as for whole HE system in Serbia.

Serbia has four types of Higher Education Institutions (HEI): universities, faculties, colleges of professional studies (applied science) and colleges of academic studies. A university is composed of Faculties, Art Academies and sometimes Research institutes. The faculties are higher education institutions which carry out academic study programs and develop scientific, vocational, or artistic work in one or more areas. Faculties and art academies may also carry out vocational study programs. Faculties and art academies conduct legal affairs under the name of the university that they belong to, and under their own name, in accordance with the statute of the university. Keeping in mind that new model of performance bases financing of HE in Serbia, that will be presented in 2019, needs to be on faculty and colleges level, the NPCT decided firstly to develop set of KPI for HE institutions, in line to better use of project PESHES capacity and developed infrastructure. The PCB appointed special national working group for indicators proposed by the NPTC and composed by representatives of all Serbian partner institutions.

First of all it was important to define key processes and selected fields at HEI in Serbia. Working group for indicator proposed and the NPTC adopted following selected fields in which will be developed KPI:

–           Teaching and learning

–           Research

–           Knowledge transfer

–           International orientation

–           Finance and infrastructure capacity

–           Student employability

–           Student satisfaction

Also, the NPTC adopted Procedure for Working Groups, Quality Planning and Control in process of developing set of KPI.

Definition of key processes and its decomposition provided the bases for development and optimization of the set of performance indicators for study programs, institutions and HIE system as whole.

As it is mentioned above the PCB appointed special national working group for indicators (WGI) proposed by the NPTC and composed by representatives of all Serbian partner institutions.

At the first session, the Working group for indicators (WGI) nominated sub-team for preparing draft text of the Set of KPI between two WGI’s sessions. After several session (in Belgrade, Novi Sad, Niš and Novi Pazar), the WGI presented first draft of the Set of KPI (key performance indicators) at selected fields in Activity 2.1 for HE institutions to all Serbian and EU partners. After collecting of feedback from all partners and after discussion sessions, the PCB, at the meeting in Torino, March 23rd 2018, adopted final vision of the set of KPI (key performance indicators) for HE institutions.

In order to improve quality assurance system it is necessary to define set of key processes for selected fields (it is important to identify specifics of field of social sciences, technical sciences, etc. for academic and applied studies, etc. ). In addition it was necessary to define set of the core indicators that will be used for performance evaluation, profiling and ranking of institution (faculties and colleagues) and consequently study programs and universities in Serbian HE systems.

PESHES working group for indicators proposes composite quantitative performance measures in which a number of different numerical parameters are combined into a single overall indicator, e.g. for faculties:

–           Indicator 1: Success rate in students recruitment

–           Indicator 2: Internationalization and international student enrolments

–           Indicator 3: Graduation rate

–           Indicator 4: Capacity for PhD studies

–           Indicator 5: Academic staff

–           Indicator 6: Research outcomes

–           Indicator 7: Non-academic staff

–           Indicator 8: Infrastructure capacity

–           Indicator 9: Financial capacity

–           Indicator 10: Student employability

–           Indicator 11: -General student satisfaction

Detailed descriptions of all indicators are presented in the Set of KPI (key performance indicators) for HE institutions.

The set of key performance indicators for HE institutions, study programmes and Serbian HE system was base for Activity 2.3. It was also necessary to provide optimization of the indicators for study programs, institutions and HE, as well as to provide process of tuning of the indicators, definition of limits and scopes as well as to define values and limit for specific indicators. Preliminary sets of meaningful indicators have to be tested against feasibility of data collection and policy acceptance.

After discussion panels in Rijeka (Croatia) and Turin (Italy) and special sessions with all stakeholders in Belgrade and Niš (Serbia), sub-team prepared draft version of the optimized set of performance indicators for study programs, institutions and HE system in Serbia. Working group for indicators in its session in Novi Sad on March 27th 2018 adopted text of optimized set of performance indicators for study programs, institutions and HE system in Serbia. Optimized set of performance indicators for study programs, institutions and HE system in Serbia will be formally adopted by Project Coordination Board (PCB) at its meeting planed for June 13th 2018 during study visit to the University of Twente, after discussion panel “Proposed Serbian Model for Performance Based Evaluation in HE System” also planed to be organized during study visit to the University of Twente.

During optimization process, Working group for indicators closely worked with the National Employment Service, the Republic Statistical Office, the Ministry of Education, Science and Technological Development, University Library “Svetozar Markovic”, Serbian National Library, Computer Centres of Serbian Universities, in line to test availability of data for proposed set of indicators in existing date-bases in Serbia. Working group defined potential source of data needed for calculation of indicators, which will be: OpenData portal of the Ministry of Education, Science and Technological Development (http://opendata.mpn.gov.rs/), Web of Science – WoS Core Collection (http://isiknowledge.com/), NaRDuS – National Repository of Dissertations in Serbia (http://nardus.mpn.gov.rs/), Current Research Information System in Serbia (http://e-cris.sr.cobiss.net), OpenData portal of the Republic Statistical Office (http://data.stat.gov.rs/), Statistical data of the National Employment Service (http://www.nsz.gov.rs), Serbian Universities Information Systems, etc.

National Project Coordination Team (NPCT) proposed National action plan for implementing performance-based evaluation of study programs, institutions and HE system as whole with methodology for collecting data and calculation of KPI.

Optimized set of performance indicators for study programs, institutions and HE system in Serbia will be formally adopted by Project Coordination Board (PCB) at its meeting planned for June 13th 2018 during study visit to the University of Twente, after discussion panel “Proposed Serbian Model for Performance Based Evaluation in HE System” also planned to be organized during study visit to the University of Twente.

After these steps in Activities 2.1-2.3 it was necessary to develop model for profiling and performance-based evaluation of study programmes and higher education institutions in Serbia, having in mind variety of needs reported in reports from WP.1 with usage of the best practice gained through analysis of EU models and demands of specific stakeholders (government, employers, students and academic community as well).

This process will be finalized through several iterations, discussions and optimizations of different parameters and indicators in order to get reliable set of performances and indicators for Serbian HEs and system. EU partners are monitoring the process and guiding Serbian national team, and they will also provide mentoring and advises during the whole process.

National Project Coordination Team (NPCT) nominated special national working group for mathematical model (WGM) composed by experts in fields of mathematics and statistics from Serbian Universities with experience in higher education evaluation, accreditation and management. WGM in cooperation with WGI developed mathematical model for profiling and performance-based evaluation of study programmes and higher education institutions in Serbia, and presented it to all project partner institutions and stakeholders in Serbian HE.

After discussion panel “Proposed Serbian Model for Performance Based Evaluation in HE System” planned for June 12th 2018 at the University of Twente, it is planed that Project Coordination Board (PCB) will adopt proposed Mathematical model on PCB meeting planned for September 21st 2018 at the Polytechnic University of Valencia.

Finally system of calculation and evaluation of performances will be developed based on specific indicators, their weight and importance.

Developed mathematical model for profiling and performance-based evaluation of study programmes and higher education institutions in Serbia will be discussed at special discussion panel “Proposed Serbian Model for Performance Based Evaluation in HE System” planned for June 12th 2018 at the University of Twente.

After the panel at the University of Twente and collecting feedback form all EU partners and stakeholders in Serbian HE system, the Project Coordination Board (PCB) will adopt final version of the Mathematical model on PCB meeting planned for September 21st 2018 at the Polytechnic University of Valencia. At the same meeting, PCB will adopt National action plan for implementing performance-based evaluation of study programs, institutions and HE system as whole with methodology for collecting data and calculation of KPI, proposed by National Project Coordination Team (NPCT).

The third step is also connected with national Action plan and have goal to develop ICT for support of stakeholders’ needs and data presentation. ICT support needs to provide the environment for performance evaluation and ranking of the study programs and institutions. The goal is to provide information for diverse needs (informed decisions), e.g. choosing study programmes, strategic planning, financing HE institution, etc. The basic principles could be stated as a development of multi-dimensional system, that will be user driven, and multi-level oriented. In addition data need to be accessible by user friendly web tool. The system will be developed to provide comparisons per university profile, dimension, indicator, discipline or used combined criteria. In order to develop and implement system with described features it is necessary to develop system for data acquisition and performance-based evaluation of HE institutions and study programs. The system will provide users profile for each University and faculty and they will be able to manage their profiles and provide data acquisition.

National Project Coordination Team (NPCT) nominated special national working group for ICT solution (WGIT) composed by ICT from Serbian Universities and with experience in data acquisition and multidimensional analysis.

WGIT starts its work on developing of ICT solution for data acquisition and multidimensional performance-based evaluation and/or ranking of study programs and HE institutions, but finalization of this activity depends on tangible output from WP.2, i.e. the optimized set of performance indicators for study programs, institutions and HE system in Serbia and the mathematical model for profiling and multi-dimensional ranking of study programs and HE institutions in Serbia.

Together with working groups for indicators and for mathematical model, WGIT defined ICT solution properties: 1) Integrated, multi-dimensional, multi-level oriented , 2) User-driven, with data accessible through a user-friendly web interface, 3) Stable and flexible in many aspects: extensible, scalable, durable, easy for maintenance, 4) Provided high performances, reliability, safety, data privacy and ownership, 5) Provided comparisons per university profiles, dimensions, indicators, disciplines, or uses combined criteria, 6) Provided user profiles for each university and faculty, allows users to manage their profiles and provided data acquisition, and 7) Tested by providing the pilot set of information. WGIT is considering two possible architecture and development approaches: (A) Designed as a „traditional“ software solution and (B) Designed as a Business Intelligence (BI) system

It was necessary to provide training for implementation of performance-based evaluation in Serbia at the EU partner institution. Study visits were organized at:

1)         University of Rijeka, January 18th-19th, 2018 on Key Performance Indicators for HE in Croatia, Criteria for Quality Assessment of the Part of the Activities of Higher Education Institutions within Universities, Methodology of external QA audit procedure in Croatia, Performance based HEI funding system in Croatia, Performance based HEI funding – contracts between Croatian HEIs and State

2)         Polytechnic University of Turin (Politecnico di Torino), March 21st-23rd, 2018 on Quality Evaluation System in Italian HE System, Data Collection System for Performance Based Evaluation in Italy, Departments Contribution to the Quality Evaluation Procedure at Italian Universities, Key Performance Indicators for Higher Education and Science: Italian Experience, and Performance Based HEI Funding System: Italian Experience

Training for implementation of performance-based evaluation in Serbia will be also organized at:

1)         University of Twente, June 11th-13th, 2018 on data acquisition and multidimensional ranking of study programs and institutions, key performance indicators (KPI) in HE, performance based evaluation: data and funding, data and judgement in Dutch quality assurance, U-Multirank for HE institutions, performance agreement in Netherlands

2)         Polytechnic University of Valencia (Universitat Politècnica de València), September 19th-21st, 2018 on key performance indicators (KPI) for evaluation in HE, performance based evaluation: data and funding and institutions, data and judgement in Spanish quality assurance

Also, staff from the Serbian Universities will be trained for usage of the system at Serbian Universities. It is important to provide knowledge and skills to the people who will be in charge for maintenance of Universities’ profiles and study programs’ profiles. The manuals for usage of the system will be developed.

It will be needed to provide presentation of the system for different target groups in order to present features of the systems, characteristics, dimensions and potential information for different target groups (students, labor market, industry, business).

Finally the system will be tested by providing the pilot set of information by one Faculty and study program per University. This will demonstrate capacity of the system and provide dissemination of information and benefits of the system for higher education institutions and system as whole. The pilot implementation will provide potential feedback in order to improve or better set the indicators.

The final stage is development of institutional background for developed system as well as development of basis for value based management in Serbian HEIs and system. It is important to stay that developed system will present ground and basis for performance-based evaluating and/or ranking of study programs and institutions as well as providing basis for value based management and funding of HEs based on performances. So in order to provide specific results of the project the comprehensive set of documents will be developed and proposed to HEs in order to enable performance measurement. On the other hand it is important to direct HEIs and to provide means that institution needs to take these set of action.

Comprehensive set of directions rules and instructions will be developed and presented to Serbian Universities and faculties. This set of documents will be used by National Universities.

In order to provide that the standards for accreditation and re-accreditation of study programs and institutions will be changed. The standard 11 (Quality) will be changed with obligation that institutions and study programs have to have system of performance measurement. The set of the documents as software as well will be presented as templates and system that could be used. So in the one cycle of re-accreditation all institutions and study programs will be directed to use performance measurement (based on developed model) in order to get accreditation. That step will cover all Serbian institutions and system as whole.

In addition the presented performance measurement system will be used as basis for development of directions for financing of Serbian higher educational systems (This direction will be defined by Serbian Ministry using project results in order to develop performance based funding model and management system for Serbian higher educational system). Strategic plan of the Ministry of Education, Science and Technological Development’s is to introduce new Law on funding higher education in Serbia and new Law on evaluation in higher education in 2019.

The project results in performance evaluation of higher educational institutions will be presented to Serbian Ministry and results will be used as ground for development of directive for funding of Serbian higher education (based on performances).

Finally the complete system will be introduced to other stake holders and parties that have interest (vocational educational system…).

Quality assurance, control, monitoring and management are essential for a high quality of project deliverables.

The quality management structure has been established at the project kick-off meeting to ensure effectiveness, decisiveness, transparency and quality of work. It involves the Project Consortium Board (PCB), the Project Coordinator and the Quality Assurance Work Group (QAWG). The members of the Quality Assurance Work Group (6 members from Serbian partner universities and 1 advisory member from EU partner institutions), selected according to the above mentioned procedure at the kick-off meeting and confirmed at the meeting of PCB held on October 20th, 2017, are:

1)         Prof. Snežana Nestić, University of Kragujevac, UKG, QAWG leader

2)         Prof. Aleksandra Jovanović, University of Belgrade, UBG

3)         Prof. Ivan Beker, University of Novi Sad, UNS

4)         Prof. Zoran Nikolić, University of Niš, UNI

5)         Prof. Irfan Fetahović, State University of Novi Pazar, SUNP

6)         Prof. Milan Milosavljević, Singidunum University, SUB

7)         Prof. Jose Miguel Carot Sierra, University of Valencia, UPV

This body is in charge of ensuring that the project activities are implemented according to the defined Work Plan and deadlines – dates when the activity milestones will be checked.

QAWG will perform continuous monitoring that all project events and outputs are organized and prepared according to the defined quality indicators and guidelines, as well as the rules of Erasmus+ CBHE program. Monitoring activity includes regular evaluation and reporting to ensure transparency and traceability, as well as the analysis of potential risks. For instance, when the project coordinator receives a review from EU or from NEO monitoring visits, PCB will together with QAWG analyze the document and provide a list of corrective actions. It will monitor the implementation of these actions and prepare a report about the implemented corrections and improvements. The Quality Assurance Working Group will prepare all documents related to Quality Control and Monitoring for PBC meetings.

To be cost-efficient, the QAWG meetings are organized as satellite event to the Project Coordination Board (PCB) and/or National Project Coordination Team (NPCT) meeting, and until now were held in:

1. Belgrade, on October 20th, 2017
2. Niš, on February 23rd, 2018
3. Turin, on March 23rd, 2018
4. Enschede, on June 13th, 2018

To be cost-efficient, the QAWG next meetings will be also organized as satellite event to the Project Coordination Board (PCB) and/or National Project Coordination Team (NPCT) meeting.

The main goal of the project is development and implementation of performance measurement mechanisms. The Quality assurance team is in charge to organize an external quality audit in order to control the indicators (of progress) as well as indicators of quality achievement (outcomes). So the Key Quality Indicators are: developed set of KPI, performance measurement model, ICT support, pilot programs and the most important changes in accreditations standards.

In order to provide full quality of the project results the external quality audit will be performed

Quality management and control as well as project management are essential in ensuring that the planned project objectives could be fulfilled with planned financial resources, human resources respecting planned times for the project and each activity. Different tools, techniques and strategies for successful project management and quality of processes, outcomes, deliverables and financial issues are used in order to achieve the highest cost effectiveness.

Quality control is provided through developed mechanism. The Quality Assurance Team had been established with the main purpose to provide and ensure continuous quality control and monitoring of the project results, outcomes as well as quality of the project management. Quality assurance team developed directions and documentation for all institutions that are participating in the project consortium. They are in charge to monitor all indicators of achievement of project results.

The satisfaction of all stakeholders is also measured. Also, satisfaction of the events’ participants is also measured through developed event evaluation forms.

The Quality Assurance Team will continue to monitor all indicators of achievement of project results and all project activities in line with adopted Quality control and monitoring plan and developed directions and documentation for all institutions that are participating in the project consortium.

The satisfaction of all stakeholders will also be measured till the end of the project. Also, satisfaction of the events’ participants will also be measured at the future project events through developed event evaluation forms.

First preventive monitoring visit by National Erasmus+ Office in Serbia took place in Belgrade on October 20th, 2017.

In order to provide full utilization and project results as well as to increase quality of the project activities the inter project coaching activities are and will be performed in order to increase quality of the project outcomes.

First Inter-project coaching activity: the HERE seminar on Global University Ranking and their Impact, followed with Round table discussion was organized with Serbian National Higher Education Reform Expert team at the Rectorate of the University of Belgrade, on November 7th and 8th, 2017.

Organization of meetings with non-academic representatives will contribute to dissemination of project results and promotion of new concept of performance measurement, in line with adopted Dissemination and Exploitation Plan

Project Consortium Board (PCB) adopted the Dissemination and Exploitation Plan on October 20th, 2017.

First dissemination activity was presentation of the PESHES project at the HERE seminar on Global University Ranking and their Impact, organized with Serbian National Higher Education Reform Expert team at the Rectorate of the University of Belgrade, on November 7th and 8th, 2017.

ERASMUS + PESHES web site was developed and published on December 05, 2016. All information about the project have been published on this web address. It contain all developed materials, reports, directions that may serve to the public in Serbia and also to institutions and public at Western Balkan Counties (at least), as well as project results, outcomes and activities. Also, web portal is used for document management for the project consortium as well.

PESHES web site was redesigned by Center for promotion of science team and new version of the site is launched.

The performance measurement, indicators and rankings have large importance for all HEs. On the other hand it is important to provide proper introduction of the new concept. In EU countries different forms of discussions and awareness campaign lasted for years before the formal introduction of the system, in line with adopted Dissemination and Exploitation Plan

Daily management of the project was done by University of Belgrade, as coordinator, under the supervision of the appointed contact person. He took care of all events, helped by the administrative staff. University of Belgrade ensured continuous coordination of all activities and adequate preparation of all the planned events. If necessary, University of Belgrade consulted Project Consortium Board and National Project Coordination Team by organizing physical and virtual meetings. In such a way, smooth running of the project was ensured.

Every day activities will be subject to the project management. The lead organization in each working package will be in charge for the specific set of activities. The reports will be developed about project progress.

The quality management structure has been established at the project kick-off meeting to ensure effectiveness, decisiveness, transparency and quality of work. It involves the Project Consortium Board (PCB), the National Project Coordination Team (NPCT), the Project Coordinator and the Quality Assurance Work Group.

PCB was established during the kick off meeting and is responsible for all management activities of the project. PCB Composition: The institutional coordinator of each beneficiary nominates a member of the Project Consortium Board. The member of PCB has the mandate to negotiate on behalf of his/her institution at each PCB meeting. If the member of PCB is unable to attend a PCB meeting, the beneficiary shall temporarily appoint a deputy. PCB Decision Making: At the each PBC meeting, no less than half of the members shall constitute quorum (50% + one member). Decision making is by simple majority among the present members (one vote per member).

In the Serbia National Project Coordination Team also was established during kick-off meeting and the NCB consist of 1 representative of all institutions from Serbian side.

Organized meetings:

1. The kick off meeting for ERASMUS+ PESHES project was held on December 12th-14th, 2016, in Belgrade, Serbia.
2. National Project Coordination Team (NPCT) meeting was held on December 26th, 2016. in Novi Sad, Serbia
3. National Project Coordination Team (NPCT) meeting was held on March 6th, 2017. in Belgrade, Serbia
4. Project Consortium Board (PCB) meeting was held in Belgrade, on October 20th, 2017
5. National Project Coordination Team (NPCT) meeting was held on November 30th, 2017, in Belgrade, Serbia
6. Coordination meeting was held in Rijeka, on January 19th, 2018
7. National Project Coordination Team (NPCT) meeting was held in Niš, Serbia, on February 23rd, 2018
8. Project Consortium Board (PCB) meeting was held in Turin, Italy, on March 23rd, 2018
9. National Project Coordination Team (NPCT) meeting was held in Novi Sad, Serbia, on March 27th, 2018

Project consortium meetings will be organized till the end of project eligibility period. The main issues will be addressed during these meetings.

The project will be subject to an external financial control. Organizational and financial project management activities are very important for the project success.

As per EACEA and Earsmus+ CBHE application guidelines, an external financial audit will be done with the aim to evaluate cost-efficiency of the project implementation. External audit will be done by an external audit company, selected in the public procurement process.

Copies and originals of all supporting relevant documents will be kept at the University of Belgrade as the grant applicant institution in five years after the end of project eligibility period.