I've analyzed multiple benchmarks for long-context LLM performance published in 2024-2025 through comprehensive scraping and search results. Here's a structured comparison of the major benchmarks:

1. Loong (2024)

• Purpose: Evaluates long-context understanding in realistic multi-document scenarios where all documents are relevant to the answer (no document can be ignored)
• Key Features:
  • Four task types: Spotlight Locating, Comparison, Clustering, Chain of Reasoning
  • Context lengths: 10k-250k tokens across 4 sets
  • Documents sourced from financial reports, legal cases, and academic papers (2024)
• Notable Findings:
  • Top models (e.g., Gemini-1.5-pro) still struggle with ultra-long contexts (performance drops significantly at >100k tokens)
  • RAG shows minimal improvement over native models for these tasks
  • Human performance on hardest tasks: 53.7% accuracy with 15-minute time limit

2. Ada-LEval (2024)

• Purpose: Length-adaptable benchmark for evaluating LLMs' long-context capabilities across varying context lengths
• Key Features:
  • Two subsets: TSort (text sorting) and BestAnswer
  • Supports context lengths up to 128k tokens
  • Designed to test models in "ultra-long" settings (100k+ tokens)
• Notable Findings:
  • Reveals significant gaps in models' capabilities at extreme lengths
  • Shows that even top models (e.g., GPT-4o) have limitations in ultra-long contexts

3. LongBench v2 (2025)

• Purpose: Assesses LLMs' ability to handle long-context problems requiring deep understanding across real-world multitasks
• Key Features:
  • 503 multiple-choice questions across 6 categories
  • Context lengths: 8k-2M words (majority under 128k)
  • Human expert validation (53.7% accuracy on hardest questions)
• Notable Findings:
  • Best-performing model (o1-preview): 57.7% accuracy (vs. human baseline of 53.7%)
  • Clear performance degradation at longer contexts (e.g., 128k+ tokens)
  • Leaderboard shows models like Gemini-2.5-Pro and DeepSeek-R1 outperform others

4. Vellum AI LLM Leaderboard (2025)

• Purpose: Comparative benchmark of LLMs' capabilities, price, and context window sizes
• Key Features:
  • Includes models from 2024-2025 (e.g., GPT-4o, Gemini-2.5-Pro, Qwen3-235B)
  • Focuses on practical metrics like context window size and real-world performance
• Notable Findings:
  • Highlights the gap between claimed context window sizes and actual performance
  • Shows that most models (even those with "1M" context windows) show performance drops beyond 128k tokens

Cross-Benchmark Comparison Summary

Critical Insights from All Benchmarks

1. Performance Decline at Scale: All benchmarks show significant performance degradation beyond 100-128k tokens, indicating that current models don't fully leverage their claimed context windows
2. Realism Gap: Benchmarks like Loong and LongBench v2 reveal that models struggle with realistic multi-document scenarios (where evidence is scattered across documents) better than synthetic benchmarks
3. RAG Limitations: RAG implementations show minimal benefit for these tasks (Loong results indicate RAG often fails to retrieve all relevant documents)
4. Model Variability: Larger models (e.g., 235B+ parameter) show better performance but still fall short in ultra-long contexts compared to human capabilities

This comparison demonstrates that while 2024-2025 benchmarks have advanced long-context evaluation, there remains a significant gap between theoretical context window capabilities and practical performance in real-world multi-document scenarios.

The differences between Jan-Nano, Lucy, and Jan-V1 can be summarized as follows based on available information:

1. Jan-Nano

   • A 4-billion parameter model developed by Menlo Research for deep research tasks.
   • Achieves 83.2% accuracy on the SimpleQA benchmark (per arXiv technical report).
   • Optimized for local deployment with tools like MCP (Multi-Channel Protocol) for web search.
   • Has variants like Jan-Nano-128k, which features a native 128k context window for improved research capabilities.

2. Lucy

   • A 1.7-billion parameter model created by Menlo Research as a lightweight, mobile-friendly variant.
   • Designed for agentic web search and lightweight browsing (e.g., using tools like Serper and Crawl4AI).
   • Benchmarked at 78.3% accuracy on SimpleQA with MCP, slightly below Jan-Nano (80.7%), but optimized for mobile/CPU efficiency.
   • Built on Qwen3-1.7B with RLVR (Reinforcement Learning with Value Functions) and "machine-generated task vectors" to enhance reasoning.

3. Jan-V1

   • A 4-billion parameter model from Menlo Research's Jan Family, explicitly built based on Lucy.
   • Achieves 91.1% accuracy on SimpleQA, outperforming Jan-Nano by ~8 percentage points.
   • Focuses on agentic reasoning and problem-solving via enhanced tool integration (e.g., for complex tasks).
   • Targets seamless integration with the Jan App for local, high-performance research workflows.

Key Distinctions:

• Size & Use Case: Lucy is the smallest (1.7B) and mobile-optimized; Jan-Nano and Jan-V1 are both 4B but Jan-V1 is a scaled-up version of Lucy for higher performance.
• Performance: Jan-V1 > Jan-Nano > Lucy in SimpleQA benchmarks (91.1% → 83.2% → 78.3%).
• Technical Approach: Lucy uses RLVR with task vectors for efficiency; Jan-V1 inherits this but scales it for better accuracy and tool utilization.
• Context Window: Jan-Nano-128k offers a larger context window (128k) than base Jan-Nano, but Lucy is designed for mobile constraints.

This hierarchy reflects Menlo Research's progression from lightweight mobile models (Lucy) to scalable research tools (Jan-Nano and Jan-V1).