Can GPT-3 Perform Statutory Reasoning?

Andrew Blair-Stanek (1, 2) , Nils Holzenberger (2) , Benjamin van Durme (2)

1 University of Maryland [Baltimore]
2 JHU - Johns Hopkins University

Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we explore the capabilities of the most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider a variety of approaches, including dynamic few-shot prompting, chain-ofthought prompting, and zero-shot prompting. While we achieve results with GPT-3 that are better than the previous best published results, we also identify several types of clear errors it makes. We investigate why these errors happen. We discover that GPT-3 has imperfect prior knowledge of the actual U.S. statutes on which SARA is based. More importantly, we create simple synthetic statutes, which GPT-3 is guaranteed not to have seen during training. We find GPT-3 performs poorly at answering straightforward questions about these simple synthetic statutes. CCS CONCEPTS • Applied computing → Law; • Computing methodologies → Natural language generation; Reasoning about belief and knowledge.

Keywords: natural language processing reasoning law statutes GPT-3; natural language processing; reasoning; law; statutes; GPT-3

Domains: Computer Science [cs]

Andrew Blair-Stanek, Nils Holzenberger, Benjamin van Durme. Can GPT-3 Perform Statutory Reasoning?. ICAIL 2023: Nineteenth International Conference on Artificial Intelligence and Law, Jun 2023, Braga, Portugal. pp.22-31, ⟨10.1145/3594536.3595163⟩. ⟨hal-04360081⟩