Evidence state: Framework. Methodology framework only. No completed HoopAI benchmark, customer result, model comparison, or empirical performance finding is reported. Illustrative examples are not observed results.
Protocol designation. This document proposes a change-control evaluation. No completed HoopAI regression run, version comparison, audit finding, or rollback outcome is reported.
The system under review includes more than a model. Prompts, sources, retrieval, tools, permissions, interfaces, and downstream actions can all change behavior.
The evidence record must follow that full package through approval and recovery.
Use the release package as the unit
Can a business team explain, test, approve, monitor, and reverse a material change to an AI-assisted workflow?
The unit is one immutable release package. It links the intended task to every configured component needed to reproduce or interpret the release.
A package should identify the model and provider setting, system and task prompts, tool definitions, source manifest, retrieval configuration, policy rules, permissions, and code revision.
It should also record the owner, reason, affected users, environment, release time, approval, evaluation evidence, monitoring window, and recovery target.
Classify the change before testing it
Define change classes by possible consequence, not by line count. A one-word instruction can alter an action boundary more than a large refactor.
- Editorial: wording changes intended not to alter facts, policy, routing, or action.
- Behavioral: prompt, model, retrieval, or tool changes expected to affect task outcomes.
- Boundary: changes to data, permissions, users, regions, or downstream actions.
- Emergency: time-sensitive containment with abbreviated approval and mandatory later review.
Each class should specify reviewers, required tests, rollout shape, monitoring, and recovery evidence. Allow escalation when uncertainty is high.
Document expected differences before the run. A change without an observable hypothesis is difficult to evaluate and easy to rationalize afterward.
Maintain three kinds of cases
Contract cases
These prove essential behavior: source boundaries, required fields, prohibited claims, permission checks, approvals, escalation, and safe failure.
A failure in a critical contract case should block release even if average quality improves elsewhere.
Regression cases
These preserve reviewed failures and representative workflows. Every case needs a reason, expected outcome, owner, and date last confirmed.
Remove obsolete cases through review, not silently. A suite can preserve yesterday's process after the business task has changed.
Fresh probes
Use unseen cases to detect overfitting to the stable suite. Sample from new source patterns, languages, edge conditions, and operational incidents.
Keep probe construction separate from the team tuning the release where resources allow. Record any exposure before final evaluation.
Compare old and proposed packages
Run both packages on the same versioned cases under matched settings. Preserve raw outputs, traces, actions, errors, latency, and reviewer decisions.
Randomize review order and mask package identity when practical. Reviewers should not assume the proposed version is better because it is new.
For nondeterministic systems, predefine repeat policy and sampling parameters. Do not rerun only failed cases until a favorable output appears.
When external systems change between runs, mark the comparison as unmatched. Exact reproduction may be impossible, but traceability should still identify known differences.
Score intended improvement and regression separately
Use the workflow rubric for task quality, grounding, policy, escalation, and action correctness. Add a change-focused label to every meaningful difference.
- Expected improvement: matches the stated hypothesis without creating a material loss.
- Neutral variation: differs but remains within acceptable output bounds.
- Regression: violates an established expectation or increases review burden.
- Unexplained change: material difference not predicted by the change record.
Track severity and task slice. A critical action regression must remain visible rather than being offset by many small style improvements.
Reviewers should record evidence for disputed changes. Independent approval is appropriate for customer-facing, data-handling, or high-consequence releases.
Exercise rollout and recovery
Test the deployment path in an isolated environment, including configuration retrieval, cache behavior, permissions, monitoring, and version display to operators.
Use shadow or limited exposure only when governance permits it. Define eligibility, duration, stop conditions, and who can expand exposure.
Trigger rollback during rehearsal. Confirm that the prior package, dependencies, and source state can be restored without leaving mixed or orphaned actions.
Measure detection and recovery evidence, not only elapsed time. Operators must be able to connect an observed event to the correct package.
Anticipate governance failures
Mutable prompts can break provenance. Unversioned source changes can alter behavior without a release. Provider updates can create drift outside the team's direct control.
Logs may omit tool state, be retained too briefly, or expose sensitive content. Rollback may restore code while leaving changed records in downstream systems.
Emergency paths can become routine bypasses. Require retrospective review, complete the missing evidence, and close the exception with an accountable owner.
Make a version-specific decision
The outcome should approve, reject, or condition one release package for named workflows. Different tasks may receive different decisions from the same change.
Publish the expected gains, regressions, unresolved changes, review burden, monitoring window, and recovery limits. Avoid saying the new model is generally better.
Reopen review when a dependency changes materially, even if the application revision does not. Configuration management includes the operating context.
Scope limits and disclosure
Stable cases cannot represent every future input. Nondeterminism and third-party drift can limit reproduction, while extensive logs can create privacy and security concerns.
NIST SP 800-53 and the SSDF provide authoritative control and development references. They do not prove that this proposed process is complete or implemented.
A later empirical report must identify package manifests, change class, cases, run policy, reviewers, deviations, regressions, rollout, rollback, incidents, and conflicts.
No completed HoopAI change-control result is contained in this framework. Its examples define possible records and do not describe current product capabilities.
Methodology
Create a change package containing task contract, prior and proposed versions, reason, owner, affected users, evaluation set, expected differences, risk classification, rollout, monitoring, and recovery. Run a stable regression set before and after the change. Use controlled exposure where consequences warrant it and preserve the evidence needed to reproduce an observed output within approved retention rules.
Score expected improvement, regression count and severity, policy adherence, action correctness, reviewer effort, and unexplained change. A critical regression should block release even if average quality improves. Require independent review for material customer-facing or data-handling changes and record the approval decision with unresolved limitations.
Limitations
- Third-party model behavior can change outside the application team’s direct control.
- Exact reproduction may be limited by nondeterminism and changing external systems.
- A regression set can preserve old assumptions after the business process changes.
- This framework does not prove that logs are complete, secure, or legally sufficient.
Sources
- NIST SP 800-53 Rev. 5, Configuration Management and Audit controls: Primary control catalog used to frame traceability, configuration, audit, and recovery questions. Control selection and assessment remain organization-specific.
- NIST SP 800-218, Secure Software Development Framework 1.1: Primary secure-development reference for release evidence and supplier questions. It is not a security assessment of HoopAI or any vendor.
- NIST AI Risk Management Framework: Voluntary risk-management reference. It informs terminology and review questions; it does not validate HoopAI or any result in this framework.
Notes
Methodology framework only. No completed HoopAI benchmark, customer result, model comparison, or empirical performance finding is reported. Illustrative examples are not observed results.