LLMs in Historical Research: Technical Constraints and Methodological Problems

Large language models are built around assumptions that often conflict with historical research. The first part of this workshop explains how these assumptions enter the system, from the encoding and tokenization of text to the choice of training data, alignment, and system prompts. We will look at how historically significant distinctions can disappear during preprocessing, how models trained on modern texts handle historical language, and how their responses are shaped by instructions hidden from the user. Sampling and the way models are deployed create further problems for reproducibility. These details matter because they determine which research questions can reasonably be investigated with a given model.

The second part is a critical examination of the Cyphersolver project (https://github.com/dbourdeau/cyphersolver), which uses agentic LLMs to decrypt unsolved historical ciphers. As a largely automatically created historical research database that touches on a variety of sources and methods from transcription, computational analysis, and proposed decipherment, it is an ideal test case to investigate the contributions of human intervention, external sources, generated code, and the language model itself, including the distinction between recovering a cipher key and locating an existing decipherment. We will consider how historical assumptions enter the analysis, whether the controls adequately test the proposed solutions, and how uncertainty is represented in the results.