Abstract / Description of output
In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.
Original language | English |
---|---|
Title of host publication | Findings of the Association for Computational Linguistics ACL 2024 |
Editors | Lun-Wei Ku, Andre Martins, Vivek Srikumar |
Publisher | Association for Computational Linguistics |
Pages | 8599-8618 |
Number of pages | 20 |
ISBN (Electronic) | 9798891760998 |
DOIs | |
Publication status | Published - 16 Aug 2024 |
Event | The 62nd Annual Meeting of the Association for Computational Linguistics - Centara Grand and Bangkok Convention Centre at CentralWorld, Bangkok, Thailand Duration: 11 Aug 2024 → 16 Aug 2024 Conference number: 62 https://2024.aclweb.org/ |
Publication series
Name | Proceedings of the Annual Meeting of the Association for Computational Linguistics |
---|---|
Publisher | Association for Computational Linguistics |
ISSN (Print) | 0736-587X |
Conference
Conference | The 62nd Annual Meeting of the Association for Computational Linguistics |
---|---|
Abbreviated title | ACL 2024 |
Country/Territory | Thailand |
City | Bangkok |
Period | 11/08/24 → 16/08/24 |
Internet address |