Turkey 2

Models of Evaluation For Research Proposals in Turkey : Models of Evaluation For Research Proposals in Turkey Prof. Dr. Omer CEBECI Vice President – Funding, TUBITAK (The Scientific and Technical Research Council of Turkey) S. GENC, A. KERC, H. KARATAS, A. FEYZIOGLU, F. COSKUN, O. OZPEYNIRCI, G. KOZANOGLU


Slide2 : SUMMARY Three-dimensional evalution used by TUBITAK for the evaluation/selection of research project proposals grouped under three categories (1) curiosity driven academic research, and (2) customer driven applied research, both in universities and research institutions, and (3) technological and innovation driven research conducted by the private industry Details of the “Phrase-anchored rating scale”


Research Proposals : Research Proposals *from universities and research institutions: curiosity driven academic research customer driven applied research *from private industry: technological & innovation driven research


Slide4 : Share of TUBITAK in TR Fiscal Budget


Evaluation Criteria : Evaluation Criteria three-dimensional evaluation criteria developed in collaboration with researchers & reviewers


Curiosity DrivenAcademic Research- the three dimensions - : Curiosity Driven Academic Research - the three dimensions - intellectual/scientific/professional merit expected impact of the anticipated outcome achievability of the research with the proposed research team, equipment/facilities and methods


Customer Driven Applied Research- the three dimensions - : research and development merit expected impact of the anticipated outcome achievability of the research with the proposed research team, equipment/facilities and methods Customer Driven Applied Research - the three dimensions -


Technological & InnovationDriven Research- the three dimensions - : technological level of the research innovative level of the product/outcome feasibility of the process Technological & Innovation Driven Research - the three dimensions -


Evaluation Process : Evaluation Process all three-criteria are given equal weights Phrase-Anchored Rating Scale versus Likert Scale sub-criteria phrases describing *very competitive *competitive *not-competitive features of the proposal


Very competitive : Very competitive *scientifically and professionally outstanding & very well justified project & *points to an opportunity for a major contribution to the advancement of the knowledge and/or to the resolution of a problem of practical importance


Competitive : Competitive *scientifically and professionally competent and justified proposal which will make a contribution to the advancement of knowledge &/or the resolution of a problem of practical value & therefore *support is suggested if funds are available


Not competitive : Not competitive *work routine in character *scientifically and professionally unsatisfactory and poorly organized


Evaluation Process : Evaluation Process evaluate 10-15 proposals by referring to the sub-criteria phrases a meeting for a final verdict Panels 5-8 Individual Reviewers


Curiosity / Merit = Not comp. : Curiosity / Merit = Not comp. Scientific consistency and the rationale of the research are not clearly explained. A clear scientific / technological question is not put forward. Project is more like an investigation / data collection / routine work. References provided include similar studies & literature survey does not point out the basis / importance of the project. The research is based on unreliable data and hypothesis.


Curiosity / Merit = V. Comp. : Curiosity / Merit = V. Comp. Outcomes have high potentials for publication in journals or books listed in international indexes The originality of the work has been supported by extensive and critical literature survey Hypothesis for evaluating the research topic is very well defined Explanation and analysis of the expected outcomes reveal the superiority of the work in comparison to the existing science and technology.


Curiosity / Impact = V. Comp. : Curiosity / Impact = V. Comp. Very important in terms of sustainable development of the country /very likely to be implemented to find solutions for the problems of society Very likely to be employed in different scientific & technological fields Very likely to generate new projects Commercialization potential of the outcomes is very high The project is supported by international, national or industrial sources.


Curiosity / Impact = Not comp. : Curiosity / Impact = Not comp. Potential for adding value to science and technology is low Subject of the project is not among the priorities of the country Not likely to result in intellectual property worth-protecting.


Curiosity / Achiev. = V. Comp. : Curiosity / Achiev. = V. Comp. Project Team The team is experienced in national / international projects related with same/similar subject They have experience as advisors /authors / referees / editors / book authors The can allocate enough time for the project Competencies / responsibilities / roles of the team members are well defined and adequate End users of the project outcomes are also members of the team.


Curiosity / Achiev. = Not Comp. : Curiosity / Achiev. = Not Comp. Project Team The team is not experienced in conducting projects of this size The knowledge and awareness of the team is not sufficient They do not have important publications in the subject of the project Some of the team members are irrelevant for the project Essential competencies are lacking


Curiosity / Achieve. = V. Comp. : Curiosity / Achieve. = V. Comp. Infrastructure Infrastructure of the institution is very adequate for the project Additional equipment requested within the scope of the project is very compatible with the existing infrastructure and the project Existing sources / equipment are used rather than purchasing new sources Requested equipment can also be used in other / future projects


Curiosity / Achieve. = Not Comp. : Curiosity / Achieve. = Not Comp. Infrastructure Infrastructure of the institution is not adequate for the project, unless supported with major equipment Equipment requested within the scope of the project is not compatible with the existing infrastructure and the project


Curiosity / Achieve. = V. Comp. : Curiosity / Achieve. = V. Comp. Methodology Approach / methodology are very well designed to reach the target Methodology is correct and well-defined; and standard methods and literature are cited Preliminary experiments have been conducted to rationalize the hypothesis Alternatives (plan B) have been considered if difficulties are encountered


Curiosity / Achieve. = V. Comp. : Curiosity / Achieve. = V. Comp. Methodology Methodology is not adequate to reach the target Relations between the experiments and hypothesis are not well defined Methodology is not explained with a common scientific basis Possible problems and limitations are not considered Statistical analytical requirements are not considered


Curiosity / Achieve. = V. Comp. : Curiosity / Achieve. = V. Comp. Timeline Proposed period & time schedule are realistic Budget Proposed budget is realistic and well-justified Project is also supported by other institutions


Curiosity / Achieve. = Not Comp. : Curiosity / Achieve. = Not Comp. Timeline Proposed period and time schedule are not synchronized Time schedule is not adequate Budget Budget is not well-define and requested amount is too low / high There is no possibility of support from other institutions


Customer / Merit = V. Comp. : Customer / Merit = V. Comp. Aims to develop a national / international novel technology (methodology, system, product, process / technique) Brings comparable superiority to the existing system Outcome will be a technology to be protected under the intellectual property rights Work consists of a scientific and rationale approach An interdisciplinary project with the collaboration of the Public- Private Industry-University-Research Institutions


Customer / Merit = Not comp. : Customer / Merit = Not comp. Literature survey and market survey are not satisfactory, work is based on unreliable data No scientific / technological rationale and integrity in the project It is a study that had been done before in the country Aim, objective and motivation are not clear Outcome is not qualified as “applicable/usable” Not related with a real need / problem


Innovation Level = V. Comp. : Innovation Level = V. Comp. Technology / product developed aims to fulfill a gap in the existing technology or replace the existing technology within the following 3 years Has an interdisciplinary approach to solve more than one problem Rationale of the R&D is well established (theoretical / analytical / experimental) Added value of the anticipated outcome of the R&D project is considered Have contribution to increase R&D staff A doctorate / masters study is incorporated within the project and the outcomes have potentials to be published in national/international journals.


Technological Level = Not comp. : Technological Level = Not comp. Literature survey and market survey are not adequate No scientific/technological advance and integrity in the project R&D rationale (analytical and /or experimental) is not adequate Aim, objective and motivation are not clear.- Procedure that has been used will not provide a change / improvement in the methodology and technology


Innovation Level = V. comp. : Innovation Level = V. comp. Outcomes may lead to spin-off company Outcomes include production standards and technical specifications Outcomes will affect other sectors as well Outcomes very likely to increase the export capacity of the country Very likely to be a basis for generating new projects / products


Innovation Level = Not comp. : Innovation Level = Not comp. Outcome is not qualified as “applicable/usable” Period of usefulness is very limited or not effective Does not provide new areas of work and as a result does not increase employment Expenditure for the R&D study is far beyond the expected economical benefit of the product Support provided by the private organization is not sufficient.


Comments : Comments Reviewers (hundreds): The model reduced the, burden of the review process on them, & subjectivity and variability of the opinions of the individuals Convenience of selecting from a comprehensive list of phrases as well as the freedom of offering their own judgments Researchers (thousands): Sub-criteria phrases to be very instrumental in guiding the development of their proposals


Thank you : Thank you Hundreds of scientists, researchers and reviewers participated in workshops for the development of the criteria and phrases listed in the tables. Their contributions are gratefully acknowledged.