Assessing Gender Awareness in Research Evaluation: A Text Analysis Approach

Speaker: Gabriel Okasa, Swiss National Science Foundation

Co-Authors:

Abstract

This analysis investigates how gender awareness is addressed in evaluation reports for the SPIRIT funding scheme of the Swiss National Science Foundation (SNSF). Gender awareness is one of seven evaluation criteria used to assess proposals. It considers both the gender composition of the research team and the consideration of sex and gender in the research content. Building on the methodological framework established by Okasa et al. (2025), this analysis expands the scope to include 1,582 evaluation reports submitted by three different types of evaluators: external evaluators, panel members and gender equality experts. While panel members received specialized training on the assessment of gender awareness, external evaluators did not.

The text classification analysis of the evaluation reports shows that evaluations within gender awareness criterion are less positive, and focusing less on suitability of methods, feasibility of the project or the track record of the applicant, as compared to other evaluation criteria. Within the gender awareness criterion, panel members and the gender equality experts write longer reviews and provide more suggestions as compared to external evaluators. In addition, they tend to be more critical than the external evaluators.

Further text mining analyses show a clear pattern for the overall assessment criterion which summarizes all evaluation criteria – reports conducted by the panel members mention “gender” in more than half of the reports, while those written by the external evaluators mention “gender” in less than a quarter of the reports. There are notable differences in the grading patterns for the gender awareness criterion based on the evaluator – the external evaluators give the highest grades, while the gender equality experts give the lowest grades, panel members are in between.

These findings underscore the importance of evaluation standards and reviewer training to ensure consistent and meaningful integration of gender awareness in research assessments.

Okasa, G., de León, A., Strinzel, M., Jorstad, A., Milzow, K., Egger, M., & Müller, S. (2025). A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports. Quantitative Science Studies, 1-40. DOI: https://doi.org/10.1162/QSS.a.23

Back to top