Snipara Benchmark: Project Memory Beats Raw Context Dumps

Snipara is not a token reducer. It is a project memory and context compiler for AI agents. This benchmark measures one visible side effect of that job: when Snipara selects the right source context before generation, the prompt becomes much smaller, easier to verify, and less dependent on raw context dumping.

The product question

Raw tokens are a weak interface for serious agent work. Agents need durable project memory, source-aware retrieval, reviewed team context, code impact context, and a consistent way to carry that knowledge across users, models, and sessions. This article isolates one part of that system: does Snipara choose a useful context package, or does it merely make prompts smaller?

The answer from this run is narrower and more useful than a generic cost-saving claim. Snipara selected a compact context package that improved measured answer quality and reduced unsupported factual claims, while avoiding the failure mode of dumping arbitrary project history into the model window.

The mistake we did not publish

The first hallucination run compared Snipara against a "without Snipara" baseline that attempted to send the full corpus to the model. That older corpus was about 601K tokens. The OpenAI model used for that first run had a 128K context window.

The result was predictable: the baseline produced context-length errors. Scoring those errors as normal answers created a misleading 0% factual accuracy number for the baseline. That number is not useful for marketing, so we excluded it.

The clean setup

The cleaner benchmark keeps the baseline inside a realistic raw context budget. The raw baseline receives the first 32K tokens of the indexed corpus with no retrieval or memory selection. Snipara uses hosted context retrieval with an 8K token budget and selects context against the task. The June 19 rerun used GPT-4.1; the full corpus fit that model's long context window, but this comparison measures focused context selection against a fixed raw window, not full-dump long-context performance.

Results

Older GPT-5.5 follow-up

We also ran a GPT-5.5 follow-up. The first attempt produced several empty answers because GPT-5 Chat Completions count internal reasoning against the completion budget, so we raised that budget and retried empty answers.

In an earlier 11-case GPT-5.5 follow-up, Snipara scored 7.91/10 versus 5.18/10 for the raw baseline, while using about 3,004 tokens/query instead of 32,000. The June 19 GPT-4.1 rerun above is now the current complete hosted benchmark.

The interesting part is the model comparison. With Snipara selecting the project context, GPT-4.1 scored 9.15/10 in the June 19 rerun, while the earlier GPT-5.5 follow-up scored 7.91/10 on the same benchmark family. In other words, the retrieval layer made a cheaper or older model competitive with a newer one for this project-context task set. Bigger models still matter, but the context compiler moved a large part of the outcome before generation started.

The useful conclusion is not that retrieval magically eliminates hallucinations in every prompt. It is that dumping or slicing raw context is an expensive relevance strategy. In the GPT-4.1 run, the smaller Snipara context also produced a lower true hallucination rate. Snipara behaves like a memory compiler: it decides which project sources should reach the model, then leaves the model to reason over a smaller and more focused evidence set.

The benchmark should still be read with the right boundary: it is a project-context benchmark, not a universal hallucination benchmark. The verifier counts direct factual errors and contradictions against the context each answer path received; omissions, unverifiable claims and wrong framing are separate quality issues.

What the claim-grounding metric showed

In the June 19 GPT-4.1 audit, Snipara's true hallucination rate was 1.2% versus 2.6% for the raw-window baseline. The verifier also measured 93.0% factual accuracy with Snipara versus 69.9% for the baseline. The remaining misses are useful follow-up cases for retrieval, answer-pack framing and benchmark prompt design.

There is an important distinction here: the claim-grounding audit measures faithfulness to the supplied context, not overall answer quality. One shared-context case scored low on answer quality even though it only had one unsupported claim, because the retrieved context can still lead the model toward the wrong framing. That is still a retrieval-quality issue, just not the same failure mode as inventing a directly false detail.

The defensible claim is still not "Snipara eliminates hallucinations." The defensible claim is narrower and stronger: in this GPT-4.1 run, Snipara gave the model a source-selected project memory package that fit the window, improved answer quality, and reduced direct factual errors versus the raw-window baseline.

The next proof point

The stronger test is a corpus that no tested model can ingest as a full dump. In that setup, the comparison is no longer Snipara versus "all evidence in the prompt." It is Snipara versus first-window truncation, chunk stuffing, and LLM compaction. Those baselines can lose facts before generation starts.

We ran an internal overflow validation for that case. It measures evidence recall before answer generation. That evidence-recall gate is important: if the context package does not contain the source facts, no answer-quality score can rescue the run.

This validation used a 1.52M-token overflow corpus with the real source documentation placed after the raw baseline window. It is not yet a full answer-quality run, but it confirms the failure mode that matters most for oversized corpora: before generation starts, raw truncation can lose every source claim.

Why this matters

For agent workflows, the practical failure mode is not only hallucination. It is wasting context budget on irrelevant project history, then asking the model to infer what matters. Snipara shifts that work into project memory: select the relevant source truth first, keep it reusable across sessions, and let the model reason over context that was chosen for the task.

That is why this benchmark matters, but also why it should not define the whole product. Token reduction is a measurable symptom. The product is project intelligence that decides what deserves to be in front of the model: docs, code graph evidence, team rules, prior decisions, and reviewed context that survives tool and model changes.