Generative AI and Higher Education Studies Workshop II: The Use of Generative AI for Science of Science and Higher Education Studies

Università della Svizzera Italiana, Lugano, February 5-7, 2025

Call for contributions

The development of generative AI language models, i.e. “a class of artificial intelligence models that can create new data based on patterns and structures learned from existing data” (Ray 2023) represents one of the most significant technological change of the recent years. The development of the models has sparked wide interest in the research community, but also in various application fields of research results. The versatility of such models, such as ChatGPT developed by OpenAI, makes them suitable for a wide range of scientific research and its application. The literature highlights both risks and challenges of the use of generative AI including increased efficiency and accuracy, but also security concerns and capability limitations (Deng and Lin 2022).

As of the impact of generative AI on research and higher education, the literature on AI focuses first and foremost on the implications for educational activities (Grassini 2023), including specifically its usage for scientific writing (Imran and Almusharraf 2023). Potential issues in terms of quality and ethical concerns (e.g., ChatGPT does not cite its sources) have been highlighted in this context (Haman and Školník 2023).

Generative AI has some attractive features for science and higher education studies, including the fact that it can access and summarize a wide range of sources available on the Internet. The interactive features of ChatGPT are potentially relevant as these allow users to interact with the AI to get more precise and relevant results (Bornmann and Lepori 2024).

In scientometrics, ChatGPT has been used in some recent studies such as to identify research topics (Tang, Zhou and Lu 2023), to predict citation counts (de Winter 2024) and to identify prominent scholars (Sandnes 2024). Results regarding the usefulness of AI in that respect are somewhat mixed, with some studies highlighting the potential of generative AI, while others arguing that the quality of results falls short of the current standards in scientometrics (Sandnes 2024). Since the evolution of language models is so rapid, these research results from the empirical studies may be outdated soon. Three other areas which have been investigated in scientometrics are the use of ChatGPT to evaluate research quality (Thelwall 2024), to identify similar institutions for benchmarking (Bornmann and Lepori 2024) and to categorize research papers based on metadata of publications such as titles and abstracts (van Eck and Waltman 2024). In these areas, we expect that the interactive nature and easy use of ChatGPT become highly attractive to (non-)expert users. Non-expert users frequently do not want to bother with the use of more complex tools requiring expert knowledge and interpretation. Expert users are interested in the potential of the new competitor.

Since many areas in scientometrics have been and will be more and more confronted with the use of generative AI, the goal of the workshop is to advance our understanding of its potential, including but not limited to ChatGPT. The workshop is intended to identify pitfalls and risks for science of science and higher education studies and to propose approaches to mitigate them.

The organizers of the workshop particularly welcome empirical studies comparing systematically the output from generative AI (e.g., ChatGPT) with more standard approaches in scientometrics, science of science studies, and higher education studies to perform tasks such as:

The labeling of research topics based on publication metadata.The identification of prominent scholars and the measurement of their performance.The evaluation of research quality at the paper, individual and institutional level.The analysis of scientific careers (e.g., the identification of certain structures).The identification of similar institutions in benchmarking studies.The characterization of universities’ missions, strategies and organization.

Workshop organization

The workshop will be organized in a set of thematic sessions around the topics introduced above. Each accepted submission will be assigned to a discussant, who will be expected to read the paper in advance and provide a detailed feedback.

Funding

A limited number of grants will be provided to cover and accommodation, particularly for young researchers.

Reviewing process

We welcome submissions presenting on-going work and including at least some preliminary results. Short papers of around 3000 words with a short abstract of 50-150 words (incl. references, appendices and other material) should be submitted by 31st August 2024. Short papers must be exclusively in English and include the name and e-mail address of the presenter and the title of the contribution.

All contributions submitted to the conference will be reviewed by at least two anonymous reviewers. Based on the outcomes of the peer review process, the conference’s program committee will select the contributions for presentation at the conference, either as an oral presentation or as a poster presentation. Submissions will be peer-reviewed and selected based on their quality.

Publication

Selected papers, as well as main conclusions of the workshop, will be published by the journal Scientometrics.

Scientific committee: Benedetto Lepori (Università della Svizzera italiana), Lutz Bornmann (Max Planck Society), Frans Kaiser (University of Twente), Wolfgang Glänzel (KU Leuven), Marek Kwiek (University of Poznan), Mike Thelwall (University of Sheffield), Marco Steenbergen (University of Zurich), Jens Peter Andersen (Aarhus University).

Organization committee: Agata Lambrechts (USI), Pınar Eldemir (USI).

Important dates:

References

Bornmann, L., & Lepori, B. (2024). The use of ChatGPT to find similar institutions for institutional benchmarking. Scientometrics, 1-6.

de Winter, J. (2024). Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts. Scientometrics, 1-19.

Deng, J., & Lin, Y. (2022). The benefits and challenges of ChatGPT: An overview. Frontiers in Computing and Intelligent Systems, 2(2), 81-83.

Grassini, S. (2023). Shaping the future of education: exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences, 13(7), 692.

Haman, M., & Školník, M. (2023). Using ChatGPT to conduct a literature review. Accountability in research, 1-3.

Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4), ep464.

Ray, P. P. (2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems.

Sandnes, F. E. (2024). Can we identify prominent scholars using ChatGPT?. Scientometrics, 129(1), 713-718.

Tang, L., Zhou, X., & Lu, M. (2023). A GPT-Based Approach for Scientometric Analysis: Exploring the Landscape of Artificial Intelligence Research. arXiv preprint arXiv:2304.09487.

Thelwall, M. (2024). Can ChatGPT evaluate research quality?. arXiv preprint arXiv:2402.05519.

van Eck, N. J., & Waltman, L. (2024, January 24). An open approach for classifying research publications. Leiden Madtrics. https://www.leidenmadtrics.nl/articles/an-open-approach-for-classifying-research-publications