From Pāṇini to LLMs: Towards a Hybrid Paradigm for Sanskrit Computational Linguistics

The efforts in Sanskrit Computational Linguistics have now been underway for more than two decades. Sanskrit occupies a unique position among both classical and modern languages because it possesses an exceptionally rich and nearly exhaustive formal grammatical tradition, most notably that of Pāṇini. The remarkable formalism of this grammatical tradition bears an intriguing resemblance to the logic of a computer program. This has provided Sanskrit Computational Linguistics with an important advantage: linguistic knowledge can be represented in an explicit, formal, and computationally tractable manner.

This advantage has manifested itself in two important ways. First, it has enabled the development of rule-based computational tools for tasks such as word segmentation, morphological analysis, and dependency parsing. Second, these rule-based systems have provided valuable linguistic knowledge and, in several cases, high-quality gold-standard data that can be used for developing and evaluating machine-learning-based approaches.

However, rule-based systems face a fundamental limitation when dealing with semantics. A grammatical rule system can generate all analyses that are compatible with the formal grammar, but it does not necessarily have the means to determine which of those analyses is semantically appropriate in a given context. Indian grammatical theories were primarily developed as formal descriptions of language for human scholars and therefore did not need to provide computational procedures for automatically resolving semantic compatibility. This is precisely where Large Language Models (LLMs) offer a new possibility. By learning patterns of meaning and usage from large corpora, LLMs can potentially contribute to resolving ambiguities that are difficult to handle through grammatical rules alone.

At the same time, LLM-based approaches introduce significant challenges. Their internal representations and decisions are largely opaque and are not readily interpretable in terms of human-understandable grammatical rules. When an LLM produces an incorrect analysis, correcting the specific error may require additional training or fine-tuning rather than a direct modification of an explicit rule. Furthermore, the choice and composition of the training corpus can introduce biases, and the resulting knowledge acquired by the model may remain outside direct human control and verification.

Despite these limitations, LLMs present an extraordinary opportunity for Sanskrit. Sanskrit textual heritage is preserved in a vast manuscript tradition, estimated in the tens of millions of manuscripts, with some recent estimates suggesting that the number may be substantially larger. Processing this enormous and largely under-digitised textual heritage presents a challenge that cannot be addressed through conventional computational methods alone. LLMs, particularly when combined with OCR and other language-processing technologies, could provide a major impetus to the conversion of manuscripts into machine-readable texts. Once such a large-scale digital corpus becomes available, it would open new possibilities for automatic summarisation, subject identification, information extraction, semantic search, and other forms of knowledge discovery.

This talk will examine the respective strengths and limitations of rule-based approaches and LLM-based approaches in Sanskrit Computational Linguistics. It will argue for a hybrid model in which the explicit, interpretable linguistic knowledge encoded in traditional grammatical rules is combined with the contextual and semantic capabilities of modern language models. Such an approach has the potential to retain human-understandable grammatical control while exploiting the ability of LLMs to learn from large-scale textual data. Sanskrit, with its highly formal grammatical tradition and vast textual heritage, therefore provides a particularly valuable domain for exploring how rule-based linguistic knowledge and modern AI can work together in the next generation of computational language processing.