Evidence state: Framework. Methodology framework only. No completed HoopAI benchmark, customer result, model comparison, or empirical performance finding is reported. Illustrative examples are not observed results.
Research status. This paper is a protocol for evaluating a knowledge workflow. No completed HoopAI retrieval benchmark, grounding rate, privacy assessment, or production result is reported.
Grounding is treated as a chain of evidence decisions. A correct final sentence does not show whether the system found the right source, respected access, or handled a conflict.
Frame grounding as a traceable chain
Can a workflow retrieve and use approved knowledge for a named task while excluding stale, restricted, and irrelevant material?
The study also asks whether the workflow escalates when its permitted corpus cannot support an answer. Helpful uncertainty is a valid outcome.
Define four observable stages: corpus state, retrieval result, generated artifact, and proposed action. Preserve evidence from each stage for diagnosis.
Create a source register
Every document should have an owner, audience, authority level, sensitivity, effective date, review date, replacement relation, and retirement rule.
A product specification, approved help article, internal note, and account record may all contain useful text. They do not have equal authority for every question.
Record the workflow allowed to use each source and the reason. Access permission alone does not mean a source is appropriate evidence for a customer answer.
Represent provenance for derived summaries and merged documents. The W3C PROV model offers a vocabulary for entities, activities, and agents, not a complete implementation recipe.
Freeze the test corpus
Export a versioned manifest of documents, checksums, metadata, permissions, and index settings used for the run. Retain it only within approved governance rules.
Without a manifest, a changed answer may be caused by content drift, indexing, retrieval, prompt logic, or model behavior, and the evaluator cannot tell which.
Sample questions from work, then perturb the evidence
Begin with task questions that have identifiable authoritative answers: product constraints, approved messaging, support procedures, or account-specific facts within permission.
Stratify by source type, sensitivity, answer length, number of required documents, recency, language, and consequence of a wrong answer.
Create controlled variants for missing, duplicated, superseded, and conflicting sources. Add near-duplicate documents with different authority or audience.
Include questions outside the allowed corpus. The expected behavior should name the gap or owner rather than import an answer from general model knowledge.
Add access-control cases using synthetic sensitive material. The purpose is to test exclusion and trace behavior without exposing real confidential records.
Run three linked evaluations
Retrieval layer
Inspect whether the relevant source appears, whether prohibited material appears, and whether rank favors current authority over superficial similarity.
Measures may include task-level evidence coverage, restricted-source exposure, superseded-source selection, and conflict retrieval. Define relevance judgments before the run.
Generation layer
Review each material proposition against the retrieved evidence. Mark support, contradiction, omission, inappropriate synthesis, and certainty beyond the source.
Check that references point to the supporting passage and remain available to the authorized reviewer. A citation label without evidentiary support is not grounding.
Workflow layer
Observe whether the answer reaches the right person, carries the correct review state, and avoids actions outside its evidence or permission boundary.
A well-grounded answer can still be unsafe if it is sent to the wrong audience or used to change a record without approval.
Use a root-cause rubric
- Content: the authoritative material is absent, unclear, stale, or internally inconsistent.
- Index: ingestion, metadata, segmentation, or permission representation is wrong.
- Retrieval: relevant evidence is missed or a distractor is preferred.
- Synthesis: retrieved evidence is altered, merged incorrectly, or overstated.
- Control: uncertainty, escalation, audience, or downstream action is incorrect.
Score severe privacy or access events independently from answer quality. A restricted-source exposure should remain visible even when the response is factually correct.
Human review and adjudication
Use source owners to establish authority and trained evaluators to apply the rubric. Separate these roles where possible so ownership does not become automatic approval.
Reviewers should see the frozen corpus state and retrieval trace. They should not fill gaps using outside knowledge when judging whether the workflow was grounded.
Adjudicate conflicts with the documented source hierarchy. If the hierarchy cannot resolve them, the expected system behavior should be escalation.
Test the maintenance loop
Repeat selected cases after adding, revising, or retiring a source. Verify that dependencies update and that the old material no longer influences the workflow.
Measure time from an approved content change to reliable availability. Also test rollback when a source update is incorrect or its permissions are misconfigured.
Assign alerts for overdue review, orphaned ownership, unresolved conflict, and dependent workflows. These signals need accountable recipients and resolution states.
Interpreting a bounded result
Report outcomes by layer, source condition, task, sensitivity, and failure severity. A single grounding percentage hides whether errors came from content or control.
A pass applies only to the manifested corpus, index, permissions, retrieval settings, model, prompt, and workflow version. It is not a claim about all knowledge questions.
Use evidence to improve source governance or narrow the task before tuning prose. Better writing cannot repair an absent source of truth.
Limitations, privacy, and disclosure
A finite corpus cannot anticipate every future conflict, permission change, or ambiguous question. Synthetic restricted data may not reproduce production access complexity.
Correct source use does not prove completeness, fairness, privacy, or security. The NIST Privacy Framework helps frame risk work but does not certify a system.
Any empirical publication should identify corpus provenance, permissions, task sampling, perturbations, system versions, rubric, reviewers, results, incidents, and exclusions.
This proposed study contains no completed HoopAI benchmark outcome. NIST and W3C sources provide methodological context and no endorsement or implementation evidence.
Methodology
Inventory sources with owner, audience, authority, sensitivity, effective date, review date, and retirement rule. Define an allowed source set for each workflow. Create tests for current facts, missing facts, conflicts, superseded pages, restricted material, near-duplicate documents, and questions outside scope. Record retrieval results and final answers separately so an evaluator can locate whether a failure began in content, retrieval, generation, or workflow logic.
Score source selection, freshness, access control, factual support, conflict handling, citation or evidence trace, and escalation. A fluent answer from the wrong source should fail. Reviewers should classify the root cause and confirm that restricted material did not appear. Run the same cases after content, index, retrieval, permission, prompt, or model changes.
Limitations
- A test corpus cannot represent every future document conflict or permission change.
- Correct retrieval does not guarantee a complete or appropriate final action.
- Source owners may disagree about authority or freshness.
- This framework does not prove that confidential or personal data cannot be exposed.
Sources
- NIST Privacy Framework 1.0: Voluntary privacy-risk reference for data mapping and governance questions. It is not evidence that a product or workflow satisfies privacy obligations.
- W3C PROV-O: The PROV Ontology: W3C Recommendation used to inform provenance concepts for sources and derived artifacts. It does not define a complete AI-grounding evaluation.
- NIST AI 600-1, Generative AI Profile: Public generative-AI risk reference. It provides no evidence of a HoopAI implementation or satisfied control.
Notes
Methodology framework only. No completed HoopAI benchmark, customer result, model comparison, or empirical performance finding is reported. Illustrative examples are not observed results.