Figure 1: Work flow and different steps involved in the EUROFLEETS evaluation procedure
Upon call closure the first step was to check if proposals meet the eligibility criteria put forward in the “EUROFLEETS2 Guidelines for Applicants for the respective calls, which was carried out by the EUROFLEETS2 Evaluation Office (EO). If proposals failed to meet the eligibility criteria they were excluded from the further evaluation process and the decision to reject the proposal iwas taken by the SRP. Following this, a so called “watchdog”, i.e. a member of the SRP, who is an expert on the respective proposal topic, was allocated to each proposal by the chair of the SRP and the EO. The idea behind this concept is that the “watchdog” accompanies the proposals he/she is responsible for throughout the different steps of the evaluation process and if the proposal is successful, even afterwards for reporting. The first task of a “watchdog” in this respect is to recommend and suggest suitable reviewers for the individual assessment of proposals. In principle the review is carried out by three individual experts for each proposal. With regard to this task the “watchdog” is supported by the EO which contacts the suggested reviewers and surveys the preparation and reception of individual assessments.
Scientific Review Panel Meetings were held in order to provide a consensus evaluation for the submitted proposals.
During these meetings, the following working principles and procedures were agreed upon and applied:
In case of conflict of interest of any of the SRP members, either being a Principal Investigator (PI) or partner on a proposal, or belonging to an institution involved in the proposal under discussion, the SRP member is requested to leave the room.
In case not all three requested reviews are available, the following routine is applied: If two reviews are available and there is agreement on the proposal evaluation results, the proposal is discussed and a final decision is made in the consensus meeting. In case two reviews deviate considerably, a third review is requested in order to obtain a final judgement on the proposal.
If only one review is available, the proposal is discussed, however a decision on this proposal is postponed until at least one more review is available. In these cases the new reviews are circulated and a final judgement is taken by e-mail.
Applicants receive a Consensus Evaluation Report (CER), which is prepared by the “watchdog” of the respective proposal, based upon the existing individual reviews and taking into account comments and judgements made during the SRP discussion. The CER shall not contain any scores and use a common Consensus Evaluation Form provided by the EO.
The EUROFLEETS Scientific Advisory Committee (SAC) closely cooperated with this supervising the full process. One member of the SAC took part at each evaluation meeting to ensure that the evaluation and selection process and the ship-time allocation were transparent and fair. The SAC also supported this WP by giving recommendations to improve the evaluation and selection procedure.
The actual evaluation of proposals was carried out in a two step process. In a first round, all proposals for which at least two reviews were available were discussed according to the scientific criteria. Further secondary criteria, like the involvement of countries with less access to marine infrastructure (though a proper definition remains open) and new user groups, the age/position of the PI, female applicants and, if applicable, the potential use of remote access by shore based scientists were also taken into account. The watchdog gave a report and commented on the received reviews for a given proposal followed by an open discussion. In a second round the proposals were assigned to categories defined previously in the Guidelines for Applicants:
A - Recommended for scheduling
B - Additional proposals
C - Not recommended
As a result of these discussions the recommended for scheduling proposals were forwarded to the EUROFLEETS2 Logistic Review Panel (LRP) as the basis for their discussion, in order to check the logistic feasibility of the cruises. The LRP meeting took place via email communication.
Through the experience gained in both EUROFLEETS project has allowed to draw a fundamental set of rules for a European system, available here.
A brochure giving an overview of the evaluation system is also available.