Abstract / Description of output
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest---including sequence continuation, infilling, and other forms of constrained generation---involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.
Original language | English |
---|---|
Pages | 1-31 |
Number of pages | 31 |
Publication status | Published - 8 May 2024 |
Event | The Twelfth International Conference on Learning Representations - Vienna, Austria Duration: 7 May 2024 → 11 May 2024 https://iclr.cc/ |
Conference
Conference | The Twelfth International Conference on Learning Representations |
---|---|
Abbreviated title | ICLR 2024 |
Country/Territory | Austria |
City | Vienna |
Period | 7/05/24 → 11/05/24 |
Internet address |