Research and a measured proof of concept

RAG for
long PDFs.

Start with hybrid search and reranking. Keep source structure. Test every extra layer.

A study for chatbots over hundreds of long PDFs. We tested one 398-page manual and separate synthetic records for scale.

Read the findings
Source documentA fact and its context398 pages
RetrieveDense search + BM25
RerankCompare 30 candidates
AnswerCite original source pages

The useful unit is evidence with enough context.

Engineering evidence, not clinical validation. Answers received model judgments and citation checks. Expert review is pending.

Reranking earned its place.

Structure-aware chunks with hybrid search and reranking found all required evidence for the 44 answerable test questions.

Fixed chunks with reranking tied that result on complete answers. Local LLM enrichment did not improve this sample.

Keep a simple baseline.

Fixed chunks already worked well. Compare each added component against that baseline.

Use lexical and semantic search.

BM25 recovered exact identifiers that dense search missed in the synthetic scale test.

Make enrichment prove its value.

Generated context changed search results, but it did not improve final answers here. This was local context, not whole-document contextualization.

Keep evidence from every search hop.

A second search can find the missing record. A later reranker can still discard it.

The design tests patterns from Anthropic’s Contextual Retrieval and Google’s work on sufficient context.

What reached the answer?

One real PDF

These results use 44 answerable test questions. Six separate questions check whether the system abstains when information is missing.

Eight ways to retrieve the same evidence

All required evidenceSupported, complete answers
The percentage axis starts at zero. A complete answer must pass a source-fidelity judgment and deterministic citation checks.

44/44 is a small-sample result.

The Wilson 95% interval is about 92%–100%, even before judge error or correlated questions. Expert-labeled production accuracy remains unknown.

All eight variants correctly abstained on the six test controls. The development split contains 17 more questions.

Gold evidence uses required source quotes. It does not list every valid alternative passage, so answer scores can exceed gold-quote coverage.

Why citations alone did not catch one wrong answer

A question asked for a review clinic’s hours. One result described the broader outpatient department’s working hours.

The model cited that passage and gave the wrong time window. The words existed, but their scope did not match the question.

Check whether a citation supports the claim. Also check whether the claim answers the question for the correct site, group, and version.

A needle is only the first test.

Synthetic records

We added unique administrative records beside the PDF chunks. Their random codes let us check exact answers without model judgments.

Exact identifiers need lexical search

18 direct cases per size
Dense searchHybrid search
Complete target evidence in the top five results. The x-axis uses a log scale. Reranking tied hybrid search on these cases.

The largest index held 50,595 passages. Hybrid search with reranking answered all 18 direct cases. Dense search answered eight.

Paraphrased questions still contain exact entity identifiers. These templates do not represent hundreds of independent PDFs or a NoLiMa benchmark.

A second search was not enough.

Some questions required an archive record, then a linked cabinet record. One search missed the second record.

Archive referenceCabinet recordUnlock code

The follow-up search found the cabinet record. Reranking both searches against the original question often discarded that record again.

Preserve the evidence chain

18 fresh two-hop cases
New identifiers and changed wording followed six development cases. Each search used its own reranking query in the final approach.

Reserve a few passages from each search hop. The POC kept three from each hop and answered all 18 fresh cases.

Literal needles passed at every tested position

Two fictional facts appeared at 5%, 50%, and 95% of contexts with 8k, 32k, or 128k tokens.

Each cell passed both cases. These 18 simple tests did not predict the failures on linked facts.

RULER and NoLiMa explain why literal matching alone tests a narrow part of long-context use.

Most time went to model calls.

The structure-aware reranked pipeline took 5.28 seconds at the median. Its p95 was 7.42 seconds.

Complete response time

Eight requests per variant
Medianp95
Two concurrent requests. Application cache disabled. Other benchmark traffic shared the account and server.

These timings include query embedding, retrieval, reranking, and the complete answer. They exclude online judging and time to first token.

Provider caching and routing were not controlled. Eight observations make p95 descriptive, not a service guarantee.

Full-document prompting remains a useful control.

The full PDF fit the selected model’s context. It passed a smaller, 12-question quality slice and had a similar median response time.

We did not test it on all 67 questions. Hundreds of long PDFs still require document routing or retrieval.

Start with an inspectable stack.

The POC used exact vector search and BM25. This kept database approximation out of the comparison.

Projects to use or compare
RoleProjectLicenseIn this study
PDF extractionPyMuPDFPage text and Markdown tablesAGPL-3.0Tested
Alternative parserDoclingCompare on scans and difficult layoutsMITNot benchmarked
Embeddings and rerankingQwen3 8B modelsHosted through OpenRouterApache-2.0Tested

The Qwen row describes two models: Qwen3-Embedding-8B and Qwen3-Reranker-8B. Both were tested.

QdrantApache-2.0

Evaluate it as the search service for a larger corpus. Check retrieval recall again after you add approximate search and filters.

RAGFlowApache-2.0

Inspect its broader document workflow for ideas. We researched this project but did not run a product comparison.

Keep the answer model replaceable. This run used OpenRouter models, so an open-source retrieval stack does not imply fully local inference.

PyMuPDF’s AGPL license differs from a fully permissive stack. Compare Docling if that distinction matters for the application.

How we tested.

The run used 67 source-anchored questions and 615 answer evaluations. No domain expert has reviewed the labels yet.

Source
A searchable, 398-page nephrology manual from 2023. No OCR or figure-understanding benchmark.
Baseline
594 fixed chunks. Windows of 350 tokens with 45 tokens of overlap, within page boundaries.
Structure
565 chunks from Markdown paragraphs and table rows. A 350-token target. The largest observed chunk had 580 tokens.
Hybrid search
Top 40 dense and lexical candidates. Reciprocal rank fusion with k=60. Rerank 30 candidates.
Context
A cap of 4,000 tokens and 10 source chunks. Generated enrichment never became answer evidence.
Enrichment
Context from the previous, current, and next page. 546 chunks received generated text. Nineteen used a deterministic fallback.
Answer checks
A separate model checked completeness and source fidelity. Deterministic checks required a matching quote and PDF page.
Controls
Eight missing-information questions, oracle source pages, and a 12-question full-document slice. Twelve seeded judge controls passed.
Model IDs and vector settings
Embedding
qwen/qwen3-embedding-8b, 1,024 dimensions
Reranking
qwen/qwen3-reranker-8b
Enrichment
anthropic/claude-haiku-5.5
Answers
openai/gpt-6.1-sol
Judge
anthropic/claude-sonnet-5.5

Query embeddings used Qwen’s instruction format. Source citations refer to physical PDF pages, starting at one.

What these results cannot establish

  • The test split supported diagnosis during development. It was not a sealed external test.
  • Questions include model drafts and agent-curated cases. A separate model rejected two incorrect premises.
  • Page groups reduce nearby overlap. Repeated protocols can still cross the split.
  • Structure-aware chunks also change parsing and breadcrumbs. This comparison does not isolate those changes.
  • A model judge can miss errors. Twelve seeded controls do not replace calibration against human labels.
  • Synthetic scale does not prove accuracy across real documents, versions, permissions, or sustained concurrent load.

Download the PDF comparison as CSV. Download all aggregate results as JSON.

What to do next.

Use the current result as a baseline. Then test the parts that this single PDF cannot answer.

  1. Review the questions and answers.

    Ask a domain owner to review references, source scope, and clinical applicability. Resolve disagreements before you score models.

    Check: Each retained case has reviewed evidence for every required fact.

  2. Build a test set from real questions.

    Collect staff questions across different documents and protocols. Include exceptions, units, tables, cross-page procedures, and missing answers.

    Check: Hold out whole documents or protocols. Freeze the test set before further changes.

  3. Compare parsers on difficult pages.

    Use scans, columns, table continuations, figures, and superscripts. Compare PyMuPDF and Docling on the same reviewed pages.

    Check: Required values, units, negations, and source regions survive extraction.

  4. Test the actual multi-PDF corpus.

    Add document, version, section, page, table, and permission metadata. Test similar procedures and conflicting versions.

    Check: Only permitted, applicable versions reach the model. All required facts reach the final context.

  5. Compare answer models and response times.

    Keep retrieval fixed while comparing OpenRouter models. Measure complete answers, false answers on missing information, and citation support.

    Check: Report time to first token and full-response p50/p95 under isolated concurrent load.

  6. Set release criteria with the domain owner.

    Agree on accuracy and response-time targets. Run the frozen test once the design is fixed.

    Check: Publish failures and confidence limits beside the score. Add each confirmed failure to the regression set.

Read the primary sources.

These sources explain the tested patterns. Our numbers describe this POC, not a reproduction of their published benchmarks.

  1. Introducing Contextual RetrievalAnthropic Engineering, 2024

    Context prefixes, BM25, dense retrieval, and reranking. Our enrichment used neighboring pages rather than the full document.

  2. The role of sufficient contextGoogle Research, 2025

    Relevant context can still lack the facts needed for a complete answer.

  3. How should I approach evaluating my RAG system?Hamel Husain and Shreya Shankar, 2025

    Score retrieval and generation separately. Validate automatic judges against human labels.

  4. Is Semantic Chunking Worth the Computational Cost?Qu, Tu, and Bao, NAACL Findings 2025

    Keep fixed-size chunking as a baseline. Complex chunking does not bring consistent gains across the paper’s tasks.

  5. Lost in the MiddleLiu and colleagues, TACL 2023

    Test where evidence appears, not only whether a model accepts a long prompt.

  6. RULERHsieh and colleagues, COLM 2024

    Go beyond one literal needle. Add linked facts and aggregation.

  7. NoLiMaModarressi and colleagues, ICML 2025

    Reduce literal overlap between the question and evidence. This is harder than exact identifier matching.

  8. Qwen3 embedding model cardQwen

    Use model-specific query instructions and supported vector dimensions.

Reading options
Text size
Data view