AI SYSTEM TESTING
Detailed discussions
Ten extended explanations with mechanisms, examples, pitfalls, and verification approaches.
Documentation and testing process
Expected results and layered oracles
Define AI outcomes before execution even when exact wording is unknown. Separate objective facts from presentation quality. For order answers, facts include order ID, owner, amount and status; presentation includes clarity and adequate explanation. In requirement analysis, facts may be source passages and genuine contradictions. Do not treat every unexpected finding as an error: it may be a valuable discovery requiring expert adjudication.
Build layered oracles. The first validates structure/types; the second exact entities, numbers and business rules; the third claim-to-source support; the fourth completeness/relevance using a rubric; the fifth prohibited actions and privacy. Safety failures must not be offset by high average style scores. Honest insufficient-data responses may pass no-answer cases.
Example: a source allows refunds within 14 days and the user asks on day 20. Expect denial of automatic refunds or escalation according to policy, a correct explanation and citation rather than one exact sentence. An assistant citing the source while saying 30 days can pass schema and citation-existence checks but must fail factual validation.
An example acceptance rubric is 2 for all critical facts correct with no unsupported claims, 1 for a correct conclusion with a noncritical omission, and 0 for a wrong conclusion or critical unsupported claim. Specify which release cases require 2. This is a teaching rubric, not a universal industry threshold. Preserve context, model, prompt and evaluator versions for every run.
Documentation and testing process
Risk planning and release reporting
Risk assessment starts with potential harm and affected people rather than every possible test. Incorrect public-document answers have a different impact from incorrect refund tool calls. Record scenarios, likelihood, impact, controls, evidence, owners and residual risks. Likelihood × impact is a useful heuristic rather than objective truth and must not hide rare catastrophic events.
Connect risks to test levels and evidence through the strategy. Unit tests can establish permission-function behaviour, integration tests ACL enforcement in retrieval, and end-to-end tests absence of forbidden documents from context and answers. Also test caches, memory and logs; one correct retrieval filter does not cover every data path.
Lead management reports with a recommendation and its limits. Example: ready for a read-only bilingual pilot with write tools disabled; 48 of 50 agreed critical scenarios passed and two blockers remain. Task success was 91% on 200 sampled queries, which does not estimate every possible request. Then show unresolved defects, slice coverage, uncertainty and owner decisions. Pass rates without denominators and risk coverage are weak evidence.
Break effort into preparation, dataset labelling, automation, execution, analysis and maintenance. Live evaluation costs include repetitions, rate-limit delays and expert review as well as runtime. Use ranges and assumptions in Agile rather than false precision. A written test does not mitigate a risk by itself; controls, successful evidence and change review are needed.
AI and ML fundamentals
Context windows and model deployment
Context budgets include instructions, history, retrieved documents, tool definitions, tool results and generation allowances according to the API. Use the selected model's tokenizer; characters/4 is unreliable for multilingual text and code. Test token-limit boundaries and whether the application rejects, trims or summarizes. Silent truncation may remove critical policies or necessary facts.
Place a fact at the start, middle and end of long-context tests; add distractors, repeated/conflicting facts and multiple turns. Measure answer correctness/source selection rather than HTTP success alone. Summary memory should preserve important constraints and provenance; deleted preferences must not return from another cache.
On-prem planning starts with workloads: model size/precision, peak concurrency, prompt/output lengths and latency objectives. Weights are only part of RAM/VRAM needs; KV caches, buffers and batching also consume memory. Quantization may help, but evaluate quality/hardware support experimentally. CPU hosting may be possible yet unsuitable for a scenario's latency needs.
Deployment tests should cover cold starts, readiness until the model is usable, load, memory pressure, cancellation, safe shutdown and fallback. Routing requires compatible output contracts and checks for unsupported parameters/capability differences. Fallback must not violate residency or send sensitive context to another provider outside the approved policy.
RAG and retrieval
RAG experiment with a numeric example
Suppose A, B and C are the three labelled relevant documents and the top three results are X, B and Y. Precision@3 is 1/3, recall@3 is 1/3 and reciprocal rank is 1/2 because the first relevant result ranks second. With no relevant result reciprocal rank is 0. Incomplete labels make recall relative to the labelled set rather than all relevant information.
Check ranking and context utilization separately. A retriever can return B while the context builder truncates the necessary paragraph, or the model can ignore an available passage. Capture retrieved IDs, actual context IDs, token budgets and final citations. Private-document snippets must not enter unrestricted trace storage.
An experiment matrix using chunk sizes 300/600 tokens, overlap 0/60, top_k 3/6 and reranker on/off has 16 combinations. Start with a baseline and one factor, then test interactions among promising settings. Report quality, latency, context size, cost and no-answer behaviour for every variant. More retrieved text may reduce quality through noise.
Use an immutable holdout and retain a previous index for rollback. Test cache invalidation after document changes, ACL revocation and embedding/index changes. Cache keys must distinguish permissions, queries and relevant configuration; TTL alone does not guarantee immediate access revocation. Enforce access on cache reads or invalidate under a defined SLA.
Agents and coordination
Agent metrics and environment state
The final text order created does not establish order creation. Expected state may require exactly one order with the correct owner, items, amount and status. An incorrect call rejected by the backend differs from a successful unauthorized write: the first reveals a policy/selection failure, the second also breaks authorization controls. Reports should expose both.
Evaluate trajectories through invariants: every write is authorized, task IDs are preserved, budgets are respected, cleanup happens and the workflow terminates. Do not require one exact path when alternative tool sequences are valid. Reference trajectories help regression but should not penalize a shorter safe path.
For 20 scenarios with 16 successes, task success is 80%. If two scenarios include unauthorized attempts, the scenario-level attempt rate is 10%. With 100 tool calls and three unsafe attempts, the call-level rate is 3%; do not mix these denominators. Report both explicitly. Zero observed unsafe successes in 20 cases does not establish zero production risk.
Measure cooperation with valid handoffs, missing/duplicated work and conflicting outputs; measure efficiency through tool counts, duration, tokens and cost per successful task. Fewer calls may be a false optimization if necessary validation is skipped. Test cancellation during writes and checkpoint resume without duplicate orders.
MCP and contract testing
A safe MCP contract
For read_order require a bounded string order_id in inputSchema and reject unknown fields if the business contract requires it. A valid ID alone is not permission: the server must check caller identity and ownership. Models must not supply arbitrary user IDs to read another person's data. Administrative access needs a separate authorized scope.
An output schema may contain order_id, status and amount with explicit currency. Keep structuredContent and serialized text consistent when both are returned. Schema validation does not prove truth; compare against fixture databases. Distinguish unknown tools/protocol issues, execution errors and transport timeouts. Not all errors are HTTP statuses; transport and JSON-RPC have their own semantics.
Compatibility matrices cover supported client/server versions and capability negotiation. Optional output additions can still break consumers rejecting extras, so additive does not always mean safe. New required inputs generally need consumer migration. Preserve schema snapshots and semantic assertions; currency-unit changes can break contracts without schema changes.
A timeout does not prove a side effect never occurred. Retry write tools only with an idempotency strategy and operation-status checks. Load tests should cover rate limits, bounded queues, resource cleanup and auth refresh. Application correlation IDs need explicit contracts; JSON-RPC IDs connect request/response but do not replace distributed traces across an agent run.
GenAI test automation
Online evaluation and release gates
Offline golden sets allow controlled comparison but do not represent the entire production distribution. Online evaluation uses representative sampling, user feedback and delayed outcome labels. No complaint does not imply correctness: users may miss errors or leave. Correlate quality signals with task outcomes, incidents and relevant user slices.
Shadow mode evaluates alternative models on consent-safe input copies without allowing writes. Canary rollouts send limited real traffic to candidates with monitoring and rollback. A/B experiments require consistent assignment, comparable populations and safeguards. Production is not a substitute for pre-release validation of high-risk behaviours.
Gates need separate dimensions: deterministic contracts, critical safety invariants, quality rubrics, retrieval quality, latency and cost. Quality thresholds depend on business risk and calibrated labels; safety violations may block release regardless of mean quality. Compare candidates with baselines rather than absolute scores alone. Mark missing evidence incomplete, not PASS.
Monitor shifts in inputs, retrieved corpora, responses and user tasks. Define degradation responses such as rollback, read-only operation, tool disabling, review escalation or dataset refresh, with owners and deadlines. Preserve lineage across code, models, prompts, policies, corpora, embeddings, evaluators and runtime configuration.
Evaluation and golden datasets
Judge calibration and statistical interpretation
Start with 30–50 diverse examples independently labelled by two experts. Resolve disagreements and clarify the rubric before judge calibration. Separate prompt-tuning examples from blind holdouts. Binary labels support a judge-versus-expert confusion matrix: TP, FP, FN and TN; precision is the correctness of positive flags and recall the detected share of real positives. Define whether positive means a defect or an acceptable answer.
Judges can show position bias, verbosity bias, self-preference and wording sensitivity. Swap pairwise answer order, present equivalent short/long answers, paraphrase rubrics and test malicious instructions inside evaluated text. The scoring system itself needs tests. Preserve concise evidence-based rationales; private reasoning is unnecessary for audits.
Compare identical baseline and candidate cases in regression. Show per-case deltas, slices and repetitions. Choose confidence intervals or paired tests for the data structure; repeated outputs for one query are not independent new users. Case-level bootstrap may be appropriate, but state methodology and assumptions.
Thresholds such as faithfulness ≥ 0.9 are not universal. Upgrades may change denominators or judge scales. Validate evaluators on known-good/bad fixtures and do not silently convert exceptions into scores of zero or one. Judge failures, system failures and quality failures require distinct statuses and responses.
GenAI security and privacy
Data privacy and governance
Inventory the data flow: user input → logging → model provider → tools → evidence storage → evaluation dataset. For each step record owners, purposes, access, retention and deletion. Apply privacy controls to traces, debug artifacts, screenshots and exported reports too. Deleting primary records is insufficient if unrestricted evidence buckets retain copies.
Pseudonymization removes direct identifiers but mapping tables, timestamps, rare job titles and locations may allow re-identification. Evaluation should use synthetic personas that do not replicate real individuals. Production samples need lawful/approved processing, minimization and review; validate legal requirements separately for the jurisdiction and use case.
Training-data and RAG leakage need different controls. RAG ACLs protect retrieved context, but model weights may contain memorized information. Do not promise model deletion after clearing vector stores alone. Investigate suspicious outputs with controlled probes, incident triage and provider support rather than collecting actual secrets for leak tests.
Responsible AI reports include intended use, limitations, affected populations, fairness criteria and appeals. Content moderation scores are only one signal. Governance defines residual-risk authority and rollout-stop conditions. Security, privacy, fairness and usability interact: overblocking one language can reduce accessibility even without safety incidents.
Responsible AI
Fairness and adversarial evaluation
Bias is not always discrimination: systematically favouring a format or language is also a skew. Fairness needs a normative criterion defining which differences are acceptable and why. For a bilingual assistant compare task success on equivalent Ukrainian/English queries while controlling translation quality, domain distribution and sample size. Do not demand identical scores on incomparable datasets.
Select decision-system fairness metrics with stakeholders. Demographic parity compares positive-decision rates; equal opportunity compares true-positive rates across groups when appropriate labels exist. Criteria may conflict, so there is no universal fairness score. Evaluate intersectional slices such as language × task type when samples support meaningful conclusions.
Adversarial testing probes failures under intentional input changes, such as injection, irrelevant context, malformed tool results or perturbations. Guardrail evaluation should measure both harmful requests allowed and benign requests incorrectly blocked. Low attack success achieved by refusing almost all users is not good product quality.
Fairness reviews should document provenance, labels, selected groups, thresholds and limitations. Synthetic datasets do not prove absence of harm to real users. Sensitive attributes require appropriate governance and purpose rather than collection just in case. Post-release review includes complaints, appeals and changing usage, not merely rerunning old tests.