Vendors often advertise huge context windows, but effective usable context is lower. RULER (NVIDIA) tests how well models retrieve and reason over information placed at different positions in long documents.

RULER findings

RULER evaluates multiple synthetic tasks (needle-in-a-haystack variants, multi-needle, variable tracking, common words, etc.) at lengths from 4K up to 128K. Results show steep degradation for many models, especially when needles are placed in the middle or when multiple facts must be tracked.

Model class4K–8K32K64K128KNotes
Top-tier proprietaryHighHighGoodModerate–GoodBest retention but still drops
Mid-tierHighModerateDropsPoorNoticeable cliff around 32–64K
Many open modelsHighLow/variableOften collapseUnreliableStrong sensitivity to position

A key insight: “supports 128K” does not mean “performs well at 128K”. Position bias (middle of context) remains common.

Practical advice

Effective context drops with length 4K–8K High accuracy 16–32K Gradual drop 64K Often cliff 128K+ Unreliable Middle-position needles hardest to retrieve for many models

VERDICT

Context length is a ceiling, not a guarantee. Measure effective recall at target length with your own data.

Sources