Large language model

A statistical model trained to predict the next piece of text, which produces fluent output without any mechanism that checks whether the output is true.

Understanding that one fact explains every failure mode that matters for analysis. A model asked for a statute section will produce something that looks exactly like a statute section, because plausible text is what it optimises for, and a fabricated citation is indistinguishable in style from a real one.

Used well it is very good at the tasks where the answer can be checked cheaply: restructuring text, extracting fields from a document you also have open, drafting a first version, and explaining an unfamiliar concept you will then verify.

Every figure it produces goes back to a primary document before it goes into anything published.