Beyond Frontier Models: Testing Lightweight LLMs for Document Processing in RAG

Structured extraction is one of the most effective ways to enhance Retrieval-Augmented Generation systems, enabling everything from metadata filtering to Graph RAG. By enriching walls of text with summaries and keywords, you create powerful retrieval capabilities. But this comes with a cost: the pre-processing overhead of running each document through an LLM before indexing.

Do you really need expensive frontier models to get good results? We evaluated how well smaller open-weight models handle extraction tasks. This article presents evaluation results for smaller variants of DeepSeek, Gemma, Llama, Mistral, Phi, and Qwen, demonstrating how to use the Pydantic Evals framework with deterministic tests and LLM-as-a-judge approaches.

Vector search and retrieval-augmented generation have emerged as critical AI workflow tools to make business data more structured and address (and amend) the disconnect between enterprise models and execution. – Bianca Lewis, executive director of the OpenSearch Software Foundation, October 3 2025, on How RAG continues to ‘tailor’ well-suited AI

Why use smaller open-weight models? #

Why bother with small open-weight models when capable LLMs are so widely available? While individual consumers rely on ChatGPT web sessions, businesses may face stricter requirements. Public sector and NGO organizations especially need to run models locally, often as a necessity rather than a preference.

Three factors drive this need:

• Data privacy and security – Sensitive documents and extracted information stay on-premise. No data leaves your infrastructure through third-party APIs. This ensures compliance with data protection regulations and internal security policies.
• Cost efficiency – API costs disappear entirely. Operating expenses become predictable and controllable. This matters especially when processing large document volumes at scale, where per-token pricing quickly adds up.
• Independence and control – No vendor lock-in limiting your options. You maintain full control over model deployment and customization. Service availability does not depend on external providers or their pricing.

These advantages make smaller models compelling for many organizations. The critical question is whether you must sacrifice accuracy to gain these benefits. Our evaluation suggests you do not.

However, there is no shortcut to finding the right model. You need systematic evaluation with your specific data and tasks. This section shares our findings from testing several models on document extraction, followed by recommendations for conducting your own evaluation processes.

How we conducted our experiments

We designed our evaluation to balance rigor with practical constraints:

• Simple prompting for comparability – A relatively short and simple prompt ensured comparable results across models. Though techniques like few-shot prompting can improve performance, we kept the baseline prompt consistent across all model tests.
• Complex schema for stress testing – The extraction schema was fairly complex and particularly challenging for the smaller models tested. This pushed models beyond simple data extraction to test their limits.
• Selective metrics – Rather than evaluating every extracted field, we picked a few metrics highlighting different aspects of extraction quality. These are explained in detail in the table below.
• GPT-4.1 baseline – A strong general-purpose model (GPT-4.1) created reference extractions as our baseline for comparison.
• PydanticAI Evals framework – PydanticAI Evals provided the infrastructure to create evaluators and run the evaluation systematically.

Picking the right metrics

The table below shows the selected evaluation metrics we used to assess extraction quality. These examples illustrate the evaluation process and do not include the complete set of fields we extracted from each document.

Key design principles:

• Character-based fuzzy matching (fuzz.ratio) for name and manufacturer fields handles punctuation and formatting variations gracefully
• Keyword coverage uses simple substring matching for efficiency and interpretability
• Release year matching is exact (no tolerance) to ensure precision in temporal data extraction
• Coverage evaluator checks for name presence as the minimum viable extraction signal
• LLM judge focuses on description field for semantic/contextual evaluation where deterministic metrics are insufficient
• All evaluators return 0.0-1.0 scores for consistent aggregation and comparison

Evals and metrics design challenges

Our approach prioritizes deterministic evaluation methods. We keep the use of LLM-as-a-judge (where another model evaluates extraction results) limited to cases where deterministic metrics are not sufficient in order to reduce ambiguity and additional evaluation costs. However, even something as straightforward as comparing product names can be tricky depending on your specific dataset. Consider these evaluation results:

Manufacturer mismatches

Though they achieve low scores, the Gemma3 results are not actually bad. After investigation, we found that the full name "Poqet Computer Corporation" was not present in the source data. GPT-4.1 expanded "Poqet" to the full corporate name and we used those results as our baseline.

This illustrates a common challenge: you rarely have ideal reference data and often must create it synthetically. The apparent discrepancies reflect the baseline model going beyond the source text, and does not reflect the evaluated model failing. The same trade-offs between strict metrics and nuanced judgment become even more pronounced when you evaluate summarization quality, which we discuss below.

Evaluation results across models #

We evaluated 13 models on structured extraction from 168 historical computing documents, comparing their outputs against a GPT-4.1-generated baseline. While this benchmark is model-generated rather than human-verified ground truth, it provides a consistent reference point for comparison. All models were run via Ollama with quantization.

Our evaluation combined deterministic metrics (name similarity, manufacturer similarity, release year accuracy, keyword coverage, alias coverage, summary length, and field coverage) with an LLM judge (moonshotai/kimi-k2-0905 via OpenRouter) scoring summary quality on three dimensions: fidelity to baseline facts (0-4 pts), comparison to baseline summary (0-3 pts), and context/narrative quality (0-3 pts). The composite scoring applies z-score normalization with these weights: Summary Quality (30%), Coverage (10%), and 12% each for other metrics. Note that z-score normalization can amplify small metric differences and the judge model significantly shapes quality scores.

Performance rankings

Evaluation Comparison (by composite score)

The composite score above weights summary quality heavily (30%), which may not reflect your priorities. The rank-based view below offers an alternative perspective by treating all metrics equally, showing each model's average rank across individual metrics, revealing which models deliver the most consistent performance regardless of how you weight the final score.

Rank-Based View (average rank across metrics; lower is better)

Key findings

Four models cluster at the top with composite scores 0.97–1.00, separated by only 3 percentage points: gpt-oss-20b (1.00), llama3.1-8b (0.98), llama3.2-3b (0.97), and qwen3-8b (0.97). A significant gap appears after rank 4, with Phi-4 (14B) dropping to 0.91. Most models excel at field extraction, where average coverage is 95%, with llama3.2-3b and qwen3-4b achieving perfect 100%, but summary quality proved universally challenging. No model exceeded 46% on judge scores (average: 32%), with the best performers being gpt-oss-20b and qwen3-8b, both at 0.46. This dramatic gap confirms that summary generation is the hardest aspect of structured extraction.

The most striking finding is llama3.2-3b's exceptional efficiency: at just 3 billion parameters, it nearly matches the 8B llama3.1's performance (0.97 vs 0.98 composite) and achieved perfect 100% success rate (168/168 vs 167/168). This challenges conventional assumptions about parameter count requirements. The qwen3-4b paradox also stands out, achieving perfect coverage and topping several metrics (manufacturer 0.87, year 0.93, keywords 0.54) yet ranking only 9th overall due to frequently cutting summaries mid-sentence, triggering zero judge scores.

Meanwhile, gpt-oss-20b won overall with 1.00 composite despite the lowest success rate (163/168), compensating with superior summary quality. Phi-4 demonstrates clear specialization: excelling at structured fields (0.98 year accuracy, 0.93 name similarity, 0.99 coverage) while struggling with narrative generation (0.31 judge score, ranking 11th on that metric).

Summary of best lightweight LLMs

Performance patterns reveal clear size-efficiency trends. In the 1-3B range, llama3.2-3b (3B) delivers elite performance rivaling much larger models, while tiny models like gemma3-1b (1B) and qwen3-1.7b struggle with consistency despite decent coverage. The 4-8B tier shows the strongest overall performance: llama3.1-8b and qwen3-8b both achieve composite scores ≥0.97, balancing coverage reliability with respectable summary quality.

At 12-20B parameters, the picture becomes more nuanced. Phi-4 (14B) ranks 5th with 0.91 composite, excelling at structured extraction but weaker on summaries. Meanwhile, gpt-oss-20b and gemma3-12b demonstrate that larger models can excel at narrative generation (judge scores 0.46 and 0.41) without necessarily improving field extraction accuracy.

Mistral-7b's poor performance (0.00 composite, last place) stands out as particularly surprising given its 7B parameter count. However, these results reflect only this specific structured extraction task with our particular prompting approach and schema. We make no claims about overall model quality, as the same models can show dramatically different results in other contexts with alternative prompts, schemas, or use cases.

Model performance is highly task-dependent, which is why systematic evaluation with your specific data remains essential. Well-tuned lightweight LLMs for local use consistently outperform poorly-optimized larger ones, making the 3-8B range the optimal choice for production use, balancing quality, efficiency, and resource requirements.

Fact-focused vs contextual summaries #

One of the most challenging tasks during extraction, and simultaneously an indicator of the model's ability to deeply understand text, was found in creating summaries of processed documents. Technically speaking, summarization goes beyond pure extraction, but in practice it is a common requirement.

Despite deliberately not specifying the desired summary style in our system instructions, we observed a clear pattern. Smaller models produced more fact-focused summaries, while larger models attempted to present broader context and better synthesize information.

Both approaches ground summaries in factual information, but differ in scope and intent. Fact-focused summaries catalog what a computer was, while contextual summaries build on those facts to explain why it mattered in the broader arc of computing history.

Note that contextual summaries are not always preferable. If you expect dry, maximally objective fact-based summaries, specify this explicitly in your prompt.

How different models handle summarization tasks #

One small model, Llama3.2-3b, performed surprisingly well for its size at creating summaries. Its descriptions consistently balanced between fact-focused and contextual approaches. Despite not being perfect, it outperformed many larger models. We expected summaries within a specific character range (100–300 characters), which proved difficult for many models, including some larger 7B models. The otherwise capable Mistral-7b performed poorly here, often returning mechanically truncated fragments cut off around 300 characters without proper editing.

This selection of examples shows significant variability in results. Meaning a much smaller model running locally can match or even exceed top-tier commercial models. Notice how Llama3.2-3B and Gemma3-12b descriptions of the Commodore 64 outperform GPT-4.1.

What about the smallest models? Does this mean models around 1B parameters cannot handle summaries? Not necessarily. These models are designed with fine-tuning in mind. Even tiny ones like Gemma3-270M can produce quality summaries after proper adaptation. A common practice with smaller models is to separate extraction and summarization into distinct steps, as they may struggle to handle both tasks simultaneously. First perform extraction, then use the structured, cleaned data for summarization with models fine-tuned specifically for the purpose.

Key findings #

Our testing revealed several patterns:

• No universal solution exists. One model may excel at a task while another, similar model fails. Everything depends on your specific use case and requirements.
• Responsiveness to prompts improves with model size. Smaller models respond poorly to prompt instructions and few-shot learning. Medium-sized models (7B) show some improvement, while larger models handle these techniques more reliably.
• Fine-tuning extends smaller model capabilities. For relatively simple tasks with limited token usage (such as processing one-page documents), you can fine-tune smaller models on consumer hardware (24GB RAM) using quantized models distributed through Ollama or MLX. These often outperform larger general-purpose models.
• Summarization is harder than extraction across all model sizes. Smaller models (2-4B and 7B) often handle basic data extraction adequately but struggle to create quality summaries. Models around 12B parameters like Gemma-12b begin to produce consistent summaries that properly account for context. The notable exception was Llama3.2-3b, which we discussed earlier.

Recommendations: run your own lightweight LLM evaluations #

Our evaluation covers a relatively small dataset and your results will vary with different documents and extraction tasks. The goal here is to demonstrate a systematic evaluation approach you can apply to your own use case. Here is how to make that choice systematically:

• Establish systematic evaluation. Without proper test cases and reproducible benchmarks, you cannot reliably compare models. Anecdotal testing will mislead you.
• Test with your actual data. These results reflect our specific documents and extraction schemas. Your data may behave differently. Build a representative test set from your real-world documents.
• Prompt engineering matters as much as model selection. How you structure your extraction prompt significantly affects accuracy across all model sizes. Invest time in prompt refinement alongside model testing.
• Consider fine-tuning for specialized needs. If your task has specific patterns or domain language, a fine-tuned smaller model may outperform a larger general-purpose one.

Structured extraction remains one of the most powerful ways to enhance retrieval augmented generation systems. The evaluation approach demonstrated here shows you can achieve this without reliance on frontier models. With systematic testing, you can confidently deploy smaller models that deliver the extraction quality your RAG pipeline needs while maintaining control, privacy, and predictable costs.