ARC@ORU Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks

About the talk

Large Language Models (LLMs) have revolutionized many aspects of our society. Many tasks encompassing document summarization or autonomous content generation can now benefit from the capabilities of LLMs. Among these, a domain in which LLMs are receiving incresing attention is that of scientific peer reviewing. Yet, usage of LLMs in this context must be done with due care: LLMs have certain blind spots which, if exploited, can lead to detrimental effects to the human requesting the service of an LLM.

In this talk, I will outline the reasons why the author of a scientific paper may want to mislead an LLM tasked to review a given paper. Based on these reasons, I will then explain ways in which one can reach their goal via "hidden prompt injections". Finally, I will discuss the results of a large-scale systematic analysis wherein we studied the impact of prompt-injection attacks against commercial LLMs (e.g., ChatGPT, Gemini). In doing so, I will also outline potential countermeasures---as well as counter-countermeasures. The takeaway is that blind reliance on LLMs for peer-review duties is strongly discouraged, and human oversight is still necessary.

Speaker

Giovanni Apruzzese is an Assistant Professor within the Department of Computer Science at Reykjavik University, Iceland; and also within the Hilti Chair of Data and Application Security at the University of Liechtenstein. He obtained the PhD in Information and Communication Technologies at the University of Modena and Reggio Emilia (Italy) in 2020. He authored over 50 peer-reviewed papers at internationally-recognized research venues. His research interests encompass a variety of themes, most of which revolve around cybersecurity and artificial intelligence, but he also appreciates topics within human-computer interaction.

His primary expertise lies in network security and in phishing detection. Giovanni also puts a lot of effort in servicing the scientific community, and he was awarded numerous recognitions for his reviewing duties in leading computer-science venues. Due to this, he has also recently engaged in researching the usage of large-language models for scientific peer reviewing.