Skip to main content
preprint arXiv preprint 2026

Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms

Adib Sakhawat, Syed Rifat Raiyan

The language a model is tested in shifts how strongly it is pulled toward the literal reading of an idiom.

BibTeX

@misc{sakhawat2026literal,
  title         = {Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on {B}engali Idioms},
  author        = {Sakhawat, Adib and Raiyan, Syed Rifat},
  year          = {2026},
  eprint        = {2610.22526},
  archivePrefix = {arXiv}
}

Description

The question

A growing class of models generates no text at all. Decision models are non-generative scorers that return a probability over options the caller supplies, and they are increasingly used as routers, classifiers and guardrails. Because they produce nothing, chain-of-thought and rationales cannot be used to inspect them. So how do you audit one?

The method

We audit three decision models entirely through interventions on their input, using expert-annotated Bengali idioms from Bangla Bagdhara. Every idiom carries both a gold figurative meaning and a gold literal gloss, so the literal reading is a ready-made, human-certified distractor that needs no new annotation.

Laya-ml Lod-lille Jev-1.13
Design Encoder scorer Decoder scorer Closed API
Backbone mmBERT Qwen3-0.6B undisclosed
Parameters 322M 0.6B n/a
Weights open open closed
Items evaluated 8,876 888 8,761

Key findings

  1. Literal capture. Adding the idiom's own literal gloss to the options draws every model toward it, accounting for 65% to 96% of each model's remaining Bengali errors.
  2. It depends on the test language. On the same paired items, every model is 1.9 to 6.4 times more captured in Bengali than in English. The commercial model's apparent resistance turns out to be a property of the language it was tested in, and capability does not predict who fails.
  3. Confidence becomes anti-diagnostic. Calibration degrades in all three models; on the two open models, gating decisions on confidence is worse than gating at random.
  4. They are confident about nothing. On a semantics-free option list, where accuracy is at chance, the best-calibrated model still reports a mean confidence of 0.82.
  5. Order effects split by architecture. The two scoring designs differ sharply in how much option order moves their scores, which matters for anyone evaluating this class of model.

Why it matters

Decision models are being placed in front of other systems as filters and routers. This paper shows that their confidence can't be trusted as a safety signal in a low-resource language, and gives an annotation-free way to test them.

Resources