Vendors often advertise huge context windows, but effective usable context is lower. RULER (NVIDIA) tests how well models retrieve and reason over information placed at different positions in long documents.
RULER findings
RULER evaluates multiple synthetic tasks (needle-in-a-haystack variants, multi-needle, variable tracking, common words, etc.) at lengths from 4K up to 128K. Results show steep degradation for many models, especially when needles are placed in the middle or when multiple facts must be tracked.
| Model class | 4K–8K | 32K | 64K | 128K | Notes |
|---|---|---|---|---|---|
| Top-tier proprietary | High | High | Good | Moderate–Good | Best retention but still drops |
| Mid-tier | High | Moderate | Drops | Poor | Noticeable cliff around 32–64K |
| Many open models | High | Low/variable | Often collapse | Unreliable | Strong sensitivity to position |
A key insight: “supports 128K” does not mean “performs well at 128K”. Position bias (middle of context) remains common.
Practical advice
- Don’t assume you can dump everything and get perfect recall.
- Keep critical info near the top or be explicit about what to retrieve.
- For RAG-like tasks, prefer chunking + targeted retrieval over blindly stuffing long docs.
- Test with your actual documents, not just synthetic needles.
VERDICT
Context length is a ceiling, not a guarantee. Measure effective recall at target length with your own data.