Skip to main content
published LREC 2026

When Words Don't Mean What They Say: Figurative Understanding in Bengali Idioms

Adib Sakhawat, Shamim Ara Parveen, Md Ruhul Amin, Tahera Khatun, Shamim Al Mahmud, Md Saiful Islam

No evaluated model exceeded 50% accuracy against 83.4% human performance.

BibTeX

@inproceedings{sakhawat-etal-2026-when,
  title     = {When Words Don't Mean What They Say: Figurative Understanding in Bengali Idioms},
  author    = {Sakhawat, Adib and Parveen, Shamim Ara and Amin, Md Ruhul and Khatun, Tahera and Mahmud, Shamim Al and Islam, Md Saiful},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  year      = {2026},
  pages     = {6870--6879},
  address   = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  doi       = {10.63317/546w2cys6m6t}
}

Description

Can large language models understand what an idiom means, rather than what its words say? For Bengali, the seventh most spoken language in the world, there was no large-scale, deeply annotated resource to even ask the question.

What we built: Bangla Bagdhara

A culturally grounded corpus of 10,361 Bengali idioms (bāgdhārā), each annotated under a 19-field schema that was established and refined through a deliberative expert-consensus process.

Dimension What is annotated
Semantic The gold figurative meaning, plus an expert literal gloss of the words
Syntactic Surface form and certified meaning-preserving variants
Cultural The cultural context the idiom draws on
Religious Religious references and connotations
Usage A gold carrier sentence showing the idiom in context

The benchmark

We evaluated 30 state-of-the-art multilingual and instruction-tuned LLMs on inferring the figurative meaning of each idiom.

Accuracy
Best of 30 LLMs below 50%
Human annotators 83.4%

No model crossed 50%. That is a critical gap in cross-linguistic and cultural reasoning, and it shows up precisely in the kind of language that mainstream English benchmarks never test.

Why it matters

  • Foundational infrastructure. Bangla Bagdhara is the largest computational resource for Bengali figurative language, released openly with its benchmark and human-evaluation protocol.
  • Reusable as a probe. Because every idiom ships with both a figurative meaning and a literal gloss, the corpus doubles as a ready-made counterfactual test. My follow-up paper, Literal Lock-In, uses exactly that to show models being pulled toward the literal reading.

Resources