Using Large Language Models for historical data processing: the good, the bad, and the bias

Large Language Models (LLMs) are extremely powerful generative machines that compress a huge amount of knowledge into a searchable embedding. These machines are great for processing multimedial data and perform many tasks at human level, but they contain subtle biases and we need techniques for probing, evaluate and control the knowledge extracted from their embedding space.

In this talk I will present three use cases of historical data processing using LLMs: one on data generation and two about data annotation.

The data generation case is the perspectivist expansion of the Seshat databank using Deepseek (China), Llama (US) and Mistral (EU) to compare proxies of LLMs' worldviews. The quality of generated data (which is possible only with models larger than 100 billion parameters) is validated against human-compiled data by means of semantic similarity metrics using proper embedding models. Results show interesting divergencies, especially on the interpretation of elite power type.

The data annotation cases are about the comparison of humans and LLMs in the classification of Structural-Demographic phases from short texts describing historical events and the LLM-driven annotation of toxic language in texts from the XIII to the XX century to understand how much it is related to the Structural Demographic cycles. In these cases we will see the inter-annotator agreement and the perplexity scores as metrics to evaluate the annotation tasks. Results show that humans disagree in data annotation mainly for biases due to their personal views and knowledge, while LLMs disagree by causal slicing, assigning different weights to causes of multi-causal events.

There are three take-home messages. 1) the good: LLMs can be used a research tool that allows replicability and scientific evaluation in digital humanities; 2) the bad: LLMs with more than 100 billion parameters are expensive, companies have access to them, universities not always; 3) the bias: yes, LLMs are biased, but there are prompting techniques to probe their internal knowledge, and a perspectivist approach should be used rather than relying on a single model to mitigate these biases.