Skip to main content
preprint arXiv preprint 2026

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim, Multazam Mahmud, Tahsan Ferdous Lihan, Md Isfak Iqbal Chowdhury, Nafis Ahnaf Jamil, Md. Atik Shahriar, Sieam Shahriare, Wasi Omar Syed, M Safwan Hasan Khan, Tabib Hassan

BibTeX

@misc{sakhawat2026temporal,
  title         = {Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives},
  author        = {Sakhawat, Adib and Sadab, Fardeen and Fahim, Tamjid Hasan and Mahmud, Multazam and Lihan, Tahsan Ferdous and Chowdhury, Md Isfak Iqbal and Jamil, Nafis Ahnaf and Shahriar, Md. Atik and Shahriare, Sieam and Syed, Wasi Omar and Khan, M Safwan Hasan and Hassan, Tabib},
  year          = {2026},
  eprint        = {2610.17402},
  archivePrefix = {arXiv}
}

Description

The question

Narratives rarely give dates. They say after the storm, while she was abroad, years earlier. Can LLMs reconstruct when events happened relative to each other from text like that?

The task

We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations (before, meets, overlaps, during, starts, finishes, equals and their converses) between every pair of events in 360 English narratives whose events are described only relative to each other. The evaluation is pre-registered, and each output supports accuracy, consistency, repair and error analyses.

Results

Over all 101,880 event pairs, with missing answers counted as wrong:

System Accuracy Macro-F1 Coverage Accuracy on answered cells
DeepSeek 0.452 0.270 0.600 0.753
GPT-6-Luna 0.350 0.215 0.481 0.728
Gemini 0.328 0.128 0.577 0.569
Qwen 0.202 0.119 0.345 0.585
Mistral 0.001 0.003 0.024 0.049
Baseline: always "before" 0.607 0.058 1.000

Key findings

  • Output format, not reasoning, sets the score. Models answer only 34.5% to 60.0% of cells, mostly because matrix rows have one or two symbols too many or too few. A recovery parser raises the best model's coverage from 0.60 to 0.84 without any new model calls.
  • The best model beats trivial baselines on macro-F1, but not the "always before" baseline on raw accuracy under the pre-registered parser.
  • Consistency and correctness are decoupled. Converse violations differ between models by two orders of magnitude, yet forcing a logically consistent answer barely changes accuracy on the cells that were answered.
  • Where models fail: they handle precedence and containment well, rarely predict overlap, and struggle most with flashbacks, where narrative order and event order diverge.

Why it matters

Reliable structured output and genuine handling of relative time are prerequisites for narrative-level temporal reasoning, and on this evidence neither is solved.

Resources