Keep a simple baseline.
Fixed chunks already worked well. Compare each added component against that baseline.
Research and a measured proof of concept
Start with hybrid search and reranking. Keep source structure. Test every extra layer.
A study for chatbots over hundreds of long PDFs. We tested one 398-page manual and separate synthetic records for scale.
Read the findingsThe useful unit is evidence with enough context.
Engineering evidence, not clinical validation. Answers received model judgments and citation checks. Expert review is pending.
Structure-aware chunks with hybrid search and reranking found all required evidence for the 44 answerable test questions.
Fixed chunks with reranking tied that result on complete answers. Local LLM enrichment did not improve this sample.
Fixed chunks already worked well. Compare each added component against that baseline.
BM25 recovered exact identifiers that dense search missed in the synthetic scale test.
Generated context changed search results, but it did not improve final answers here. This was local context, not whole-document contextualization.
A second search can find the missing record. A later reranker can still discard it.
The design tests patterns from Anthropic’s Contextual Retrieval and Google’s work on sufficient context.
These results use 44 answerable test questions. Six separate questions check whether the system abstains when information is missing.
The Wilson 95% interval is about 92%–100%, even before judge error or correlated questions. Expert-labeled production accuracy remains unknown.
All eight variants correctly abstained on the six test controls. The development split contains 17 more questions.
Gold evidence uses required source quotes. It does not list every valid alternative passage, so answer scores can exceed gold-quote coverage.
A question asked for a review clinic’s hours. One result described the broader outpatient department’s working hours.
The model cited that passage and gave the wrong time window. The words existed, but their scope did not match the question.
Check whether a citation supports the claim. Also check whether the claim answers the question for the correct site, group, and version.
We added unique administrative records beside the PDF chunks. Their random codes let us check exact answers without model judgments.
The largest index held 50,595 passages. Hybrid search with reranking answered all 18 direct cases. Dense search answered eight.
Paraphrased questions still contain exact entity identifiers. These templates do not represent hundreds of independent PDFs or a NoLiMa benchmark.
Some questions required an archive record, then a linked cabinet record. One search missed the second record.
The follow-up search found the cabinet record. Reranking both searches against the original question often discarded that record again.
Reserve a few passages from each search hop. The POC kept three from each hop and answered all 18 fresh cases.
The structure-aware reranked pipeline took 5.28 seconds at the median. Its p95 was 7.42 seconds.
These timings include query embedding, retrieval, reranking, and the complete answer. They exclude online judging and time to first token.
Provider caching and routing were not controlled. Eight observations make p95 descriptive, not a service guarantee.
The full PDF fit the selected model’s context. It passed a smaller, 12-question quality slice and had a similar median response time.
We did not test it on all 67 questions. Hundreds of long PDFs still require document routing or retrieval.
The POC used exact vector search and BM25. This kept database approximation out of the comparison.
| Role | Project | License | In this study |
|---|---|---|---|
| PDF extraction | PyMuPDFPage text and Markdown tables | AGPL-3.0 | Tested |
| Alternative parser | DoclingCompare on scans and difficult layouts | MIT | Not benchmarked |
| Embeddings and reranking | Qwen3 8B modelsHosted through OpenRouter | Apache-2.0 | Tested |
The Qwen row describes two models: Qwen3-Embedding-8B and Qwen3-Reranker-8B. Both were tested.
Evaluate it as the search service for a larger corpus. Check retrieval recall again after you add approximate search and filters.
Inspect its broader document workflow for ideas. We researched this project but did not run a product comparison.
Keep the answer model replaceable. This run used OpenRouter models, so an open-source retrieval stack does not imply fully local inference.
PyMuPDF’s AGPL license differs from a fully permissive stack. Compare Docling if that distinction matters for the application.
The run used 67 source-anchored questions and 615 answer evaluations. No domain expert has reviewed the labels yet.
qwen/qwen3-embedding-8b, 1,024 dimensionsqwen/qwen3-reranker-8banthropic/claude-haiku-5.5openai/gpt-6.1-solanthropic/claude-sonnet-5.5Query embeddings used Qwen’s instruction format. Source citations refer to physical PDF pages, starting at one.
Download the PDF comparison as CSV. Download all aggregate results as JSON.
Use the current result as a baseline. Then test the parts that this single PDF cannot answer.
Ask a domain owner to review references, source scope, and clinical applicability. Resolve disagreements before you score models.
Check: Each retained case has reviewed evidence for every required fact.
Collect staff questions across different documents and protocols. Include exceptions, units, tables, cross-page procedures, and missing answers.
Check: Hold out whole documents or protocols. Freeze the test set before further changes.
Use scans, columns, table continuations, figures, and superscripts. Compare PyMuPDF and Docling on the same reviewed pages.
Check: Required values, units, negations, and source regions survive extraction.
Add document, version, section, page, table, and permission metadata. Test similar procedures and conflicting versions.
Check: Only permitted, applicable versions reach the model. All required facts reach the final context.
Keep retrieval fixed while comparing OpenRouter models. Measure complete answers, false answers on missing information, and citation support.
Check: Report time to first token and full-response p50/p95 under isolated concurrent load.
Agree on accuracy and response-time targets. Run the frozen test once the design is fixed.
Check: Publish failures and confidence limits beside the score. Add each confirmed failure to the regression set.
These sources explain the tested patterns. Our numbers describe this POC, not a reproduction of their published benchmarks.
Context prefixes, BM25, dense retrieval, and reranking. Our enrichment used neighboring pages rather than the full document.
Relevant context can still lack the facts needed for a complete answer.
Score retrieval and generation separately. Validate automatic judges against human labels.
Keep fixed-size chunking as a baseline. Complex chunking does not bring consistent gains across the paper’s tasks.
Test where evidence appears, not only whether a model accepts a long prompt.
Reduce literal overlap between the question and evidence. This is harder than exact identifier matching.
Use model-specific query instructions and supported vector dimensions.