AI SYSTEM TESTING
Senior
Strategy, architecture, metrics, security, and risk decisions.
01 / 12
Documentation and testing process
How do FRD and TRD affect test planning?Senior
Answer
If FRD means Functional Requirements Document, it defines behaviours and business rules; a Technical Requirements Document describes interfaces, architecture and constraints. Agree on abbreviations first. Derive scenarios, test levels, dependencies and NFR checks. Resolve contradictions and untestable requirements before implementation.
Primary sources
How should progress and results be reported to management?Senior
Answer
A status report covers completed/planned work, blockers and forecast; a completion report covers coverage, results, residual risks and release recommendations; an incident report covers impact and actions. PO/management need risks and decisions; engineering needs evidence and diagnosis. Show denominators, timeframes, baselines, critical failures, flakiness, latency/cost and sample uncertainty.
Primary sources
What should a test strategy explain?Senior
Answer
A strategy connects quality goals and product risks to testing levels, types, and techniques. It defines scope and exclusions, environments, data, automation approach, roles, evidence, entry and exit criteria, reporting, and responses to findings. It is a decision model, not a document template for its own sake.
- An AI strategy also defines model/prompt/data change policy, live-evaluation budgets, judge calibration, nondeterminism handling, and hard safety gates.
Primary sources
Examples
- For a payment feature, the strategy may place most contract checks at API level, retain a few critical UI journeys, and add production signals.
Practice exercises
- Create a one-page strategy for a change with three major risks and explain what you will deliberately not test.
Which testing metrics help, and which mislead?Senior
Answer
A metric is useful when it supports a specific decision and has a clear denominator and context. Trends in escaped defects, feedback time, or coverage of defined risks may expose a problem. Counts of test cases, defects, or automation percentage without context easily become vanity metrics and encourage harmful behaviour.
- For AI, show denominators and uncertainty separately for task success, factual errors, groundedness, unsafe actions, latency/cost, and user/data slices; one aggregate score does not establish readiness.
Primary sources
Examples
- A growing test count proves little when the suite becomes slower, flakier, and still misses critical regressions.
Practice exercises
- Choose three team metrics and document the decision, limitation, and possible gaming behaviour for each.
03 / 12
AI and ML fundamentals
How are training and overfitting or underfitting validated?Senior
Answer
Separate train/validation/test while preventing leakage, duplicates and temporal overlap. Overfitting means good training performance but poor generalization; underfitting means failure to learn necessary patterns. Appropriate capacity, regularization, data quality, early stopping and tuning can help. Check independent test sets, slices and production shifts.
Primary sources
What belongs in an effective guardrail framework?Senior
Answer
Include a risk inventory, policies, scoped permissions, input/context screening, output/schema checks, action authorization, human escalation, audit, telemetry and incident handling. Define thresholds and false-positive costs. Test model + tools + data together, version policies and monitor drift; a moderator does not replace access control.
Primary sources
04 / 12
Prompting and tool calling
How are hallucinations reduced and where is tool calling used in workflows?Senior
Answer
Use authoritative context, bounded tasks, abstention, structured outputs and independent validators; measure factual errors. Tools can search documents, read orders or delegate subtasks to agents. Delegation may use tool calls, but agent selection may also use deterministic routing. Tool output can be stale, wrong or injected.
Primary sources
05 / 12
RAG and retrieval
How are retrieval and generation failures separated?Senior
Answer
Run retrieval-only evaluation and test the generator with curated contexts too. Missing relevant passages suggest retrieval/ingestion issues; present passages with wrong answers suggest context construction or generation issues. Add no-answer, conflict, multilingual and ACL slices. Track quality, freshness, latency/cost and coverage independently; faithfulness does not establish source truth.
Primary sources
06 / 12
Agents and coordination
How are cooperation, competition and human oversight managed?Senior
Answer
Cooperation needs a shared goal and clear handoffs; competition/debate can produce alternatives, but consensus does not prove correctness. Define authority, escalation, approval for high-impact actions and cost budgets. Test that delegation cannot bypass approval; humans must see the concrete action and evidence.
Primary sources
How are complex agent behaviour and unexpected situations tested?Senior
Answer
Evaluate task completion, correct tools/arguments, state changes, termination, recovery and forbidden actions. Inject unavailable tools, empty context, contradictory data, cancellation, repeated calls and budget exhaustion. Validate final state rather than final text alone. Keep sanitized traces; unsafe actions are hard failures even if the final answer is correct.
Primary sources
07 / 12
MCP and contract testing
How are interoperability and observability tested?Senior
Answer
Test version negotiation, capabilities, supported transports, content types and auth flows. Logs/traces link runs, requests and tool executions; an application correlation ID is not a universally mandatory MCP field. Define its propagation in an instrumentation contract. Check sanitized arguments, durations, errors, retries and absence of secrets.
Primary sources
How are MCP load and chaos tests designed?Senior
Answer
In a sandbox inject latency, disconnects, partial replies, worker restarts and concurrency. Define workload, capacity, error budgets, recovery targets and stop conditions. Check cancellation, connection/resource cleanup and no duplicate writes. Repeat scenarios for non-deterministic failures; provenance and timings aid reproduction beyond average latency.
Primary sources
09 / 12
GenAI test automation
How are multi-agent workflows and CI evaluations automated?Senior
Answer
Test task graphs, roles, handoffs, tool arguments, termination and final environment state. PR gates include contracts, deterministic tests and sampled live evals; nightly runs cover broad quality/security suites. Version models/prompts/data/tools and retain per-case evidence. Hard-gate privacy/unsafe actions and compare aggregate quality with approved baselines.
Primary sources
How are guardrail tests and online evaluation integrated?Senior
Answer
Controlled datasets should include direct/indirect injection, unsafe requests and benign near-misses. Check blocked actions, leakage and false refusals. Online evaluation uses privacy-safe samples, telemetry, user feedback and delayed labels; shadow evaluators must not execute extra side effects. Monitoring complements offline gates and rollouts need rollback criteria.
Primary sources
10 / 12
Evaluation and golden datasets
Which metrics apply to multi-agent and multiturn MCP assistants?Senior
Answer
Task success = successful scenarios / evaluated scenarios; unauthorized-action rate = offending scenarios / evaluated scenarios. Also track termination, valid handoffs, correct tools/arguments, retries, recovery and latency/cost. Capture events and final state; custom code evaluators compute flags/rates. Judges assess communication quality; labels allow alternative valid trajectories.
Primary sources
How is golden data created for a personal assistant chat?Senior
Answer
Use fictitious personas, consent-safe tasks and isolated tools for scheduling, search, clarification and preference changes. Specify history, time/timezone, tools/data, expected state changes and forbidden actions. Include conflicting dates, memory deletion and cross-user attempts. Per-turn and final labels must not contain real private data.
Primary sources
11 / 12
GenAI security and privacy
How is privacy protected when deploying an LLM?Senior
Answer
Minimize data and define purpose/retention, provider data flows, encryption and access scopes; check processing terms and residency requirements. Redact logs/eval samples and control deletion/backups. On-prem hosting does not guarantee privacy: admin access, logs, telemetry and tools may leak. Threat-model the whole workflow.
Primary sources
How is leakage from training data reduced?Senior
Answer
Exclude unnecessary secrets/private records from training; use data review, deduplication, privacy-aware training where appropriate and memorization probes on approved synthetic canaries. Output filtering is an additional control, not a guarantee. Removing RAG records does not remove learned information from weights; retraining/unlearning needs separate validation.
Primary sources
What is data poisoning and how is it prevented?Senior
Answer
Poisoning deliberately introduces harmful data into training, feedback or knowledge corpora. Check provenance, ingestion permissions, review, anomalies and integrity; version datasets and maintain rollback. Test altered documents, poisoned feedback and false references in a sandbox. Content validation does not prove every source fact true.
Primary sources
Why do embeddings not replace permissions?Senior
Answer
Vector similarity measures relevance; access must be checked separately before context reaches the model.
Primary sources
Examples
- A semantically similar document belonging to another customer does not become an authorised source.
Practice exercises
- Design retrieval with tenant filtering and test a cross-tenant access attempt.
12 / 12
Responsible AI
How is continuous impact evaluation organized and which tools help?Senior
Answer
Define intended use, harm hypotheses, user slices, metrics, incident channels and review cadence. Compare trends with baselines and inspect complaints, drift and unintended use; assign remediation owners. NIST AI RMF helps structure governance; Azure AI Content Safety can provide content controls. No tool score certifies fairness or responsible operation.