Skip to main content
accepted MRL Workshop @ EMNLP 2026

Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

Fardeen Sadab, Adib Sakhawat

Output length alone explains 78.7% of between-system variance in human preference; matching outputs by length reverses the leaderboard.

BibTeX

@inproceedings{sadab2026emojis,
  title     = {Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure},
  author    = {Sadab, Fardeen and Sakhawat, Adib},
  booktitle = {Proceedings of the MRL Workshop at EMNLP 2026},
  year      = {2026},
  eprint    = {2609.29445},
  archivePrefix = {arXiv}
}

Description

When a benchmark says one LLM generates better emoji summaries than another, is that a property of the models, or of how the benchmark measured them?

The audit

We took a multilingual affective-generation benchmark apart: eight instruction-tuned LLMs producing emoji summaries for 17,100 sentences in Bangla, English and Hindi, judged in 6,960 human ratings. Our contribution is not a new emoji resource but a measurement protocol.

What we found

Claim in the benchmark What the audit shows
Systems differ significantly Treating annotators as a random factor, no system differs from any other (p = 0.76), though the conventional analysis flags 19 of 28 pairs
Ratings reflect system quality Annotator identity explains far more variance than system identity; dropping any single annotator changes the winner
The leaderboard ranks affect Mean emoji count explains 78.7% of between-system variance; a length-matched comparison over 2,599 pairs reverses the leaderboard
Providers differ in anisotropy The difference vanishes under mean-centring
Languages differ in token cost The sign flips with the normalising unit
Macro-F1 under row-wise splits Inflated by 3.1 points, and the top system changes

In short, the headline conclusions were artefacts of the instrument, driven by verbosity and annotator variance rather than system quality.

What we propose instead

Emoji–affect decodability (EAD): a reference-based probe that asks whether the affect can be recovered from the emoji. Its rankings are stable to ±0.003 macro-F1 across seeds, and it needs no annotator panel.

Why it matters

Small-panel preference evaluations are common in low-resource NLP, where annotators are scarce. This paper shows how easily they can produce confident, wrong leaderboards, and gives a cheap alternative.

Resources

  • Paper: arXiv:2609.29445 · accepted at the MRL Workshop, EMNLP 2026
  • First author: Fardeen Sadab