MAJA DOLINAR, GIULIA MALAGUARNERA, ANGELIKI TZOUGANATOU, PAOLO MANGHI, THANASIS VERGOULIS & LEONIDAS PISPIRINGAS ‣ short talk, session 3
A key element of Reforming Research Assessment (RRA) practices is to move away from simplistic approaches based on bibliometric indicators and consider a wide variety of research activities. Existing large-scale data collections cover a broader range of research output such as publications, data, software, research methods, patents, data management plans and tools. The current challenge is the lack of a robust, high-quality open infrastructure.
The presentation will explore how the OpenAIRE Graph (2024), a core service of OpenAIRE and the European Open Science Cloud (EOSC), can be utilized to address key challenges in research assessment. OpenAIRE, a non-profit partnership organisation, operates an e- infrastructure, the OpenAIRE Graph, that compiles metadata and links from over 140,740 data sources using advanced AI tools. The OpenAIRE Graph aggregates millions of metadata records from trusted data sources like OpenDOAR, re3data.org, DOAJ, and pre-print servers. Using the OpenAIRE Guidelines and PROVIDE’s metadata validation mechanism, the data undergoes a thorough deduplication process with PIDs, followed by enrichment with records from Crossref, Unpaywall, ORCID, Microsoft Academic, PubMed, DataCite, OpenCitations, and UsageCounts. AI tools enhance this metadata further by adding information on authors, classifications, Open Access status, and relationships like co-authorship and citations. This results in a Scientific Knowledge Graph of over 272 million research products, including publications, research data, and software, all linked to their funding sources and producing entities.
Whereas projects like SciLake and OSTrails have the aim to foster the quality, the FAIRness, and the interoperability of Open Infrastructures like the OpenAIRE Graph, GraspOS aims to federate open infrastructure for RRA to serve EOSC and its users. Moreover, OpenAIRE is responsible for the provision of the EOSC Knowledge Graph, used in the EOSC EU Node (2024), which aggregates metadata records and semantic links from the OpenAIRE Graph, the EOSC Service Catalogue, and other relevant European Commission databases and publishing platforms, including Open Research Europe (ORE). Through the EOSC Knowledge Graph, researchers can discover, navigate, and monitor an up-to-date global map of science, encompassing valuable information about EOSC research products, such as publications, data, and software. This enhances the capability for comprehensive research assessment on a European scale.
The proposal will highlight the OpenAIRE Graph’s and EOSC Knowledge Graph’s role in enhancing RRA by providing an open infrastructure as a possible option to proprietary databases and by enabling institutions to conduct thorough and unbiased research assessment. We will discuss the primary role of the Institutional Repositories and CRIS to provide and curate the metadata, as well as the OpenAIRE Graph’s and EOSC Knowledge Graph’s workflow, including its comprehensive aggregation-enrichment-deduplication process to ensure robust data quality and facilitate bibliometric analysis through citation metrics and indicators. Collaborative approaches to improve the coverage and the value of the OpenAIRE Graph includes collaboration with other initiatives and infrastructures such as OpenCitations and OpenAPC that joined the OpenAIRE Catalogue. OpenCitations leverage the OpenAIRE Graph data to enhance bibliometric information by integrating open bibliographic and citation data, while OpenAPC provides detailed publication cost data (APC and BPC), useful in monitoring the publications cost and the researcher behaviours towards publication’s venues.
The OpenAIRE MONITOR (2024) service provides analysis and statistics from the OpenAIRE Graph and is curated by researchers, research supporting staff, and developers. It showcases dashboards with dynamic visualisations of Open Science activities, research outputs and performance, including Research Impact, funding and collaborations indicators on demand. This service facilitates the data elaboration at institutions, university alliances or networks, research initiatives, and research funding organisations. The MONITOR service is instrumental for research performing organizations and research funding organizations empowering them to track the adoption of Open Science practices, discern the evolution of Open Access pathways over time, and evaluate associated metrics. It can also be used for creating National Monitors for Countries that want to track progress towards open access. In the proposal, we will showcase the Irish National Open Access Monitor (2024) as a prime example of how the OpenAIRE Graph supports national- level research assessment initiatives.
The efforts of OpenAIRE and its services are strongly aimed at enhancing features and indicators to align with the research assessment reforms suggested by CoARA (2022) and DORA (2024), and reaffirming its commitment to the Barcelona Declaration (2024), offering a comprehensive infrastructure that can be used to monitor and assess Open Science in research practices. To further address these topics, we participate in the CoARA Working Group (WG) “Towards Open Infrastructures for Responsible Research Assessment” to undertake a comprehensive redefinition and evaluation of essential infrastructural components for equitable research assessment. This WG is focused on establishing foundational principles and delineating the critical components required for an open infrastructure suitable for research assessment. These discussions are paramount for developing a robust framework that fosters transparent, collaborative, responsible and equitable research evaluation practices.
To this end, MONITOR’s incorporation of coverage is essential, as it bolsters the dimensions of scientific knowledge production, encompassing the rigor and integrity of research quality, the collaborative nature of open research methodologies, and the broader societal impact of scholarly work (Di Donato, 2024). For that reason, the MONITOR service adeptly integrates qualitative and quantitative indicators, providing a holistic perspective on the breadth and diversity of research endeavours. This integration advocates for the formulation of inclusive policies that recognize and reward diverse scholarly contributions, thereby influencing research assessment frameworks. The MONITOR service accomplishes this by delivering data on the openness, findability, and FAIRness (Findable, Accessible, Interoperable, Reusable) of research outputs, thus enhancing OS initiatives. In addition, integrated qualitative evaluations enable the assessment of whether a researcher exhibits specialization within a particular Field of Science or demonstrates interdisciplinary expertise across multiple domains. Specific indicators aligned with the United Nations Sustainable Development Goals (SDGs) are also integrated. These indicators are meticulously crafted to classify and analyse research contributions addressing pivotal global challenges such as climate change, biodiversity loss, pollution, and poverty reduction. By synthesizing qualitative and quantitative metrics, this approach ensures that research assessment transcends mere quantitative evaluation to encompass the broader impact and societal contributions of research.
By integrating qualitative and quantitative indicators, the MONITOR service offers a comprehensive and sophisticated approach to research assessment. This framework underscores the significance of evaluating the broader implications and contributions of research to societal and global challenges, fostering a deeper and more impactful understanding of research activities. Furthermore, the MONITOR service significantly augments infrastructure readiness by promoting discourse on cost efficiency, sustainability, and best practices. Rather than prioritizing ranking mechanisms, it fosters collaborative efforts through the OpenAIRE framework, thereby enabling universities and research-performing organizations to disseminate and adopt successful strategies. Initiatives such as University Alliances Gateways and Institutional Monitors exemplify this approach, facilitating the exchange of knowledge and fostering the development of robust, sustainable, and cost-effective research infrastructures.
Ultimately, this proposal aims to provide practical strategies for leveraging the OpenAIRE Graph and the EOSC Knowledge Graph to support informed decisions on Open Science policies and practices, fostering a more transparent, efficient, and equitable research assessment process.
KEYWORDS
monitoring; indicators; OpenAIRE Graph; open infrastructures; Open Science policies; reforming research assessment
REFERENCES
