Evidence state: Framework. Methodology framework only. No completed HoopAI benchmark, customer result, model comparison, or empirical performance finding is reported. Illustrative examples are not observed results.
Protocol status. This paper specifies a study that has not been run. No completed HoopAI sales benchmark, customer trial, revenue result, or model comparison is reported.
The proposal is meant to support one bounded decision: which parts of a named sales task may be assisted, which require approval, and which should remain outside the system.
The decision comes before the score
A useful evaluation begins with an operational choice. Examples include allowing draft call briefs, suggesting a lead route, or extracting commitments from an approved conversation record.
The study should not begin with a general question such as whether an AI model is good at sales. That framing hides differences in evidence, authority, timing, and error cost.
Primary research question
For a defined sales task, can the assisted workflow create a grounded and usable output while obeying source limits, approval states, and escalation rules?
A secondary question asks where failure concentrates. The answer may differ for routine records, sparse histories, conflicting notes, multilingual input, or requests for unsupported commitments.
Set the unit of evaluation
The unit should be a complete task case, not an isolated prompt. Each case includes its input record, permitted evidence, expected work product, action boundary, and review outcome.
Write a task contract before choosing a prompt. Name the user, business purpose, authoritative fields, forbidden inferences, required output, and person accountable for the next action.
For a call brief, the contract might permit approved account notes and product documentation. It might prohibit guessed budget, inferred intent, or claims taken from an unverified page.
Keep drafting and acting as separate conditions. Producing a proposed route is different from changing ownership in a CRM, even when both begin with the same evidence.
Construct the case set around sales reality
Sample tasks from the workflow definition, not from whatever examples are easiest to collect. Cover the account stages, regions, languages, and record qualities relevant to the decision.
Use governed historical material only when permissions and retention allow it. Otherwise build synthetic cases that preserve the structure of the task without imitating a real customer.
The set should contain routine cases, boundary cases, and cases where abstention is correct. Include missing fields, old notes, duplicate contacts, contradictory owners, and ambiguous consent.
Add cases with tempting but disallowed evidence. This tests whether the system respects the source boundary when a plausible answer exists outside the approved record.
Separate a development set from a sealed review set. The development set supports iteration; the sealed set reduces the chance that prompt tuning merely memorizes familiar patterns.
Version every case and record why it belongs. A coverage table should show which task, input condition, risk, and expected action each case represents.
Write the rubric before any run
The rubric should score dimensions independently. A fluent brief must not conceal an invented account fact, and a safe abstention should not fail because it lacks polished prose.
- Evidence fidelity: every material statement is supported by an allowed source.
- Task completeness: required fields and next-step context appear without invented filler.
- Decision control: the output remains within the permitted draft or action state.
- Uncertainty: missing or conflicting evidence is stated at the point where it matters.
- Escalation: the case reaches the correct owner when the contract requires review.
Define critical failures before scoring. An unauthorized outbound message, fabricated commercial term, silent consent override, or incorrect record change should not be averaged into a favorable total.
Human review procedure
Train at least two reviewers on the task contract and anchor examples. They should first score independently, then resolve material disagreement against written rubric language.
Record disagreement rather than smoothing it away. Repeated disputes may reveal an ambiguous policy, an incomplete source hierarchy, or a sales process that is not ready for automation.
Reviewers should see the same evidence boundary available to the system. A second pass can reveal hidden information to diagnose whether the failure came from missing context or poor use.
Observe failure families, not just averages
Tag failures as unsupported fact, omitted constraint, wrong entity, stale evidence, overconfident language, incorrect route, unauthorized action, or missed escalation.
Keep severity separate from frequency. One rare account reassignment can matter more than several weak subject lines, depending on reversibility and downstream exposure.
Measure reviewer edits by reason. A high acceptance rate can still hide extensive factual repair if reviewers silently correct outputs before marking the task complete.
Turn the run into a release decision
Report counts for every task slice, the case-set version, reviewer procedure, system configuration, and all critical failures. Do not publish only a blended score.
The interpretation should name the allowed boundary. Evidence might support assisted drafting for routine briefs while rejecting autonomous routing or any external communication.
Compare the workflow with its current human process when the decision concerns adoption. That comparison should use the same task definition and quality standard.
Re-run the sealed cases after changes to the model, prompt, retrieval logic, source set, permissions, or connected actions. A past pass does not transfer automatically.
Limits of the proposed study
Synthetic cases may miss the ambiguity, social context, and uneven maintenance of live CRM records. Historical cases may preserve past operating choices that should now be challenged.
Reviewer judgments can vary by market and sales process. A bounded set cannot estimate effects on pipeline, win rate, retention, customer sentiment, or revenue.
The protocol does not establish security, privacy, legal compliance, or fitness for every sales use. Those reviews require their own evidence and accountable owners.
Publication disclosure
This is a preregistration-style framework for future evaluation. Any later results paper must identify the sample, exclusions, versions, rubric, reviewer roles, deviations, findings, and conflicts.
The linked NIST materials inform risk and human-oversight questions. They do not certify this method, validate HoopAI, or imply that any described control is currently implemented.
Methodology
Write a task contract before selecting a model or prompt. Define allowed inputs, prohibited sources, required facts, acceptable uncertainty, intended output, action boundary, and reviewer role. Assemble versioned cases from synthetic or properly governed material. Include routine, incomplete, conflicting, stale, multilingual, and out-of-policy cases. Keep a sealed review set so prompt changes are not tuned only to familiar examples. Record model, prompt, retrieval, source, and workflow versions for every run.
Use a rubric with separate dimensions for source fidelity, factual completeness, unsupported claims, uncertainty handling, workflow correctness, policy compliance, and escalation. Two trained reviewers should independently score a meaningful sample and resolve disagreements with a documented rule. A severe prohibited action or invented account fact should remain visible as a critical failure rather than being averaged away by good writing scores.
Limitations
- Synthetic cases may not reproduce the ambiguity and drift in live CRM data.
- Reviewer judgment can vary across sales processes, languages, and regions.
- A passing result applies only to the recorded model, prompt, source, permission, and workflow versions.
- This framework does not measure pipeline, win rate, revenue, or customer sentiment.
Sources
- NIST AI Risk Management Framework: Voluntary risk-management reference. It informs terminology and review questions; it does not validate HoopAI or any result in this framework.
- NIST AI RMF Appendix C, Human-AI Interaction: Public reference for human roles and oversight. It is background material, not validation of this proposed protocol.
- NIST AI 600-1, Generative AI Profile: Public generative-AI risk reference. It provides no evidence of a HoopAI implementation or satisfied control.
Notes
Methodology framework only. No completed HoopAI benchmark, customer result, model comparison, or empirical performance finding is reported. Illustrative examples are not observed results.
