Test Retrieval Quality, Permissions,…

Direct answer

Test a grounded assistant in layers. First test whether retrieval returned the right evidence. Then test whether permission filters excluded evidence the user must not see. Then test whether only current, authoritative versions were eligible. Finally test whether the answer used the retrieved evidence correctly. A polished answer cannot compensate for a broken earlier layer.

Retrieval quality, authorization, and freshness solve different problems. Relevance asks, “Does this chunk answer the question?” Permission asks, “May this user retrieve it?” Freshness asks, “Is this version valid now?” Keep separate pass/fail signals so a high relevance score never overrides access or validity.

Build a labelled retrieval set

Create test cases before tuning. Each case should contain:

the user query;
the simulated user or role;
the evaluation date or policy version;
relevant source IDs or an explicit NONE label;
forbidden source IDs;
expected result state: answer, abstain, or escalate;
notes explaining why.

Use real query shapes, not only clean questions written by the developer. Include abbreviations, typos, short mobile queries, exact codes, paraphrases, Roman Urdu, and multi-part requests. Add deliberately unanswerable questions. If every test has an answer, your system can appear accurate while learning to answer everything.

A useful starter set has at least 30 cases: 12 normal answerable questions, five paraphrases, three exact identifier searches, three unanswerable questions, three stale-version traps, two unauthorized requests, and two bilingual or Roman Urdu variations. Expand it from production failures once real users consent to appropriate logging and review.

Metrics in plain language

Recall at k asks: for how many answerable queries did at least one correct source appear in the top k results? If the correct policy is absent, generation cannot cite it.

Precision at k asks: what share of returned results were actually useful? Low precision stuffs the context with distractions and conflicts.

Mean reciprocal rank rewards placing the first correct result near the top. A correct chunk ranked sixth may technically satisfy recall but still lose influence to five misleading chunks.

Also measure control metrics:

permission leakage rate: forbidden results returned divided by authorization test cases; target zero;
stale retrieval rate: superseded sources returned as active evidence; target zero;
unanswerable abstention rate: unanswerable cases correctly refused;
citation support rate: material claims genuinely supported by their cited excerpts;
conflict detection rate: conflicting active sources trigger review rather than blending.

Do not collapse all of these into one average. A 95% relevance score does not excuse one confidential-document leak.

Worked build: a multi-branch operations library

Suppose a retailer has public return guidance, staff operating procedures, finance-only refund limits, and separate Karachi and Islamabad delivery rules. Each document has role, branch, valid_from, valid_to, version, and owner attributes.

Test case:

{
  "query": "Can I approve a PKR 40,000 cash refund in Islamabad?",
  "user": { "role": "store_associate", "branch": "islamabad" },
  "as_of": "2026-07-17",
  "relevant": ["refund-sop-v4#approval-limits"],
  "forbidden": ["finance-controls-v2#executive-limits"],
  "expected": "ESCALATED"
}

The associate may retrieve the staff SOP showing that the amount needs approval, but not the finance-only executive controls. The correct response is an escalation, even if the forbidden document contains a more detailed answer.

Use a small harness to evaluate the retrieval function independently:

function evaluateCase(test, hits) {
  const ids = hits.map((hit) => hit.chunk_id);
  const leaked = test.forbidden.some((id) => ids.includes(id));
  const found = test.relevant.some((id) => ids.includes(id));

  return {
    query: test.query,
    permissionPass: !leaked,
    recallPass: test.relevant.length === 0 ? ids.length === 0 : found,
    returned: ids,
  };
}

for (const test of tests) {
  const filters = policyFilters(test.user, test.as_of); // server-owned
  const hits = await search(test.query, filters);
  console.log(evaluateCase(test, hits));
}

policyFilters must use authenticated server state. Do not let the model decide whether a user is finance staff, and do not retrieve everything before asking the model to hide forbidden text. Once unauthorized content reaches the generation context or logs, the boundary has already failed.

Make freshness operational

Freshness is more than adding a date field. Define a lifecycle:

A named owner approves a source.
Ingestion records checksum, version, effective dates, and indexed time.
Publishing the new version atomically changes which version is active.
The old version remains archived for audit but is excluded from normal retrieval.
A scheduled check identifies sources past their review date.
Deletion or revoked access removes the searchable copy and associated caches.

Test time boundaries: the minute before a policy activates, the minute after, open-ended validity, missing dates, and two sources accidentally marked active. If active sources conflict, fail closed and alert the owner.

Freshness monitors can compare the source registry with the index: active source count, checksum, last indexed timestamp, and chunk count. A green web page means little if the index silently missed yesterday's policy update.

Tune with evidence, not vibes

When recall is weak, inspect whether the problem is query language, chunk boundaries, missing synonyms, poor OCR, filters, or ranking. When precision is weak, inspect overly broad chunks, duplicate versions, missing metadata, and low score thresholds. Change one variable, rerun the same labelled set, and record the result.

Do not tune against the final test set until it becomes a memorized demo. Keep a development set for iteration and a held-out set for release checks. Add new production failures to a regression suite, but remove or anonymize personal content first.

Failure cases to force

A finance user switches branches and receives cached results from the previous branch.
A public FAQ and confidential SOP share a title, and the wrong one is cited.
An old document has more keyword overlap and outranks the active version.
A source update is uploaded, but indexing is still processing when traffic switches.
A Roman Urdu query retrieves no evidence despite an obvious English policy match.
An unanswerable query retrieves topically related chunks and the model guesses.
The test set uses only developer-written English and misses actual mobile phrasing.
Logs store full queries containing phone numbers or account details without a defined need.

🇵🇰 Pakistan Angle

Permissions in a Pakistani organization are often informal: “everyone in this WhatsApp group can see the sheet” is not a durable access model. Convert actual responsibilities into named roles, authenticate users, and keep the source owner accountable. Branch, franchise, campus, province, and employment type may all affect which rule applies.

Freshness is especially important where circulars and price lists arrive as scanned notices or chat attachments. Record who approved the update; do not automatically treat the newest timestamp as the highest authority. Build test queries from common English and Roman Urdu phrasing, including local product codes and currency notation. Keep evaluation files small enough for a modest laptop and separate the automated test run from live API calls where possible, so a slow connection does not prevent basic regression testing.

Hands-on deliverable

Produce a retrieval-control test pack for your cited assistant:

at least 30 labelled cases with query, role, date, relevant IDs, forbidden IDs, and expected state;
an automated retrieval harness that reports recall, permission leakage, stale retrieval, and abstention;
a source-version lifecycle and one atomic update procedure;
a permission matrix for at least three roles;
a freshness monitor specification with alert owner and response step;
before-and-after results for one retrieval improvement;
a regression file containing every failure you fixed.

Completion rubric

I evaluate retrieval before evaluating final prose.
My labelled set includes unanswerable, unauthorized, stale, multilingual, and exact-code cases.
Permission filters are derived from authenticated server state and run before retrieval.
Relevance cannot override authorization or version validity.
Active-source updates and index readiness have an explicit transition.
I report control failures separately rather than hiding them in an average score.
Test and log data are synthetic, minimized, or appropriately anonymized.

Key takeaway: A grounded system is only trustworthy when the right evidence is retrievable, forbidden evidence is impossible to retrieve, and obsolete evidence cannot masquerade as current truth.

Test Retrieval Quality, Permissions, and Freshness

Turn this lesson into one checked practice output