AI SYSTEM TESTING
Middle
Implementation, diagnosis, comparison, and failure-path testing.
01 / 12
Documentation and testing process
How should CI/CD suites be structured?Middle
Answer
PR: unit, schema, prompt rendering, mocked tools and small critical evals. Merge: integration with real test dependencies. Nightly: broader golden sets, robustness and repeated runs. Release: risk slices, security, load and human review. Production: monitoring/canary. Define separate budgets and gates.
Primary sources
How can regression be optimized in a large project?Middle
Answer
Map changes to risks and coverage, remove duplicates, isolate flaky tests, separate fast and costly suites and isolate parallel data. Retain critical smoke and periodic full runs. For AI cache only where it cannot hide model changes; version cache keys and datasets.
Primary sources
Which test techniques are useful for AI?Middle
Answer
Use equivalence classes for languages/roles/tasks, boundaries for context and tool limits, decision tables for policies, state transitions for agents and pairwise for configurations. Metamorphic tests check expected relations; adversarial tests probe robustness. Property-based tests check invariants; justify every relation.
Primary sources
How do bias and fairness testing differ from adversarial testing?Middle
Answer
Bias is systematic skew; fairness assesses whether group differences are justified under a defined criterion. Adversarial testing deliberately perturbs inputs to induce undesired behaviour. They overlap but have different objectives. Fairness metrics and groups must match the task context.
Primary sources
What makes a test case genuinely useful?Middle
Answer
A useful test case has a clear purpose, controlled preconditions and data, unambiguous actions, an observable expected result, and a link to a risk or requirement. It should be detailed enough for its audience without restating the obvious or hiding the oracle in long prose.
- For an AI case, record versioned input/context, acceptable alternatives, prohibited claims, the rubric, model/prompt/data versions, and required evidence.
Primary sources
Examples
- “Test payment” does not define success; expectations for order state, charge, event, and retry behaviour make the check diagnostic.
Practice exercises
- Edit a weak test case: remove incidental UI detail and add data, an oracle, and postconditions.
What information belongs in a good defect report?Middle
Answer
A report should let others understand, reproduce, and assess the problem: a factual summary, version and environment, preconditions and data, minimal steps, actual and expected results, reproducibility, impact, evidence, and related artefacts. Severity describes impact; priority describes urgency.
- For an AI defect, add the prompt, model, retrieval snapshot, parameters, and repeatability without storing secrets or unauthorised personal data.
Primary sources
Examples
- Instead of “payment is broken,” a useful summary names condition, location, and outcome: “Repeated POST creates a second charge after response timeout.”
Practice exercises
- Rewrite a vague bug report so another person can reproduce the problem without verbal context.
Why should severity and priority remain separate?Middle
Answer
Severity estimates the scale and nature of a defect’s impact on the product, user, or data. Priority represents business repair order considering deadlines, reach, workarounds, and cost. High severity does not always mean first to fix, while a low-severity issue can be urgent in its business context.
- A low AI-failure frequency does not reduce the severity of data leakage or an unauthorised action; assess harm separately from reproducibility.
Primary sources
Examples
- A rare crash in an admin tool may have high severity but lower priority; a homepage typo before a campaign can show the opposite pattern.
Practice exercises
- Provide four high/low severity and priority combinations and justify each with context.
What do entry, exit, and suspension criteria provide?Middle
Answer
These criteria guide decisions rather than automatically guaranteeing quality:
- Entry criteria define when a test activity is ready to start.
- Exit criteria define the evidence needed to finish it or make a decision.
- Suspension criteria define when continuing would produce unreliable results or waste resources.
- AI entry and exit criteria should name versioned datasets, the oracle/rubric, approved thresholds, repeated runs, and owners of critical residual risks.
Primary sources
Examples
- If the environment is unstable and half of the API requests fail because of infrastructure, a suspension criterion prevents the team from drawing false conclusions.
Practice exercises
- Define entry, exit, and suspension criteria for a short regression cycle and name the owner of exceptions.
How does risk-based testing help set priorities?Middle
Answer
The team connects testing depth and order to the likelihood and impact of a problem while considering change complexity, defect history, data criticality, observability, and recovery cost. It is not a magic formula but a transparent way to discuss constraints.
- AI risk analysis accounts for oracle ambiguity, labelling effort, model/provider changes, repeated runs, access boundaries, and harm from incorrect tool actions.
Primary sources
Examples
- A small invoice-format change may carry high risk when many customers need that document for tax reporting.
Practice exercises
- Build a risk matrix for three release changes and name one signal that would make you revise each assessment.
02 / 12
Python for automation
How are venv, pytest and unittest used?Middle
Answer
venv isolates Python packages from other projects; it is neither a container nor an OS sandbox. unittest is in the standard library; pytest offers fixtures, parametrization and plugins. Separate mocked unit tests from live AI evaluation so PRs do not depend on provider latency/cost.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
What are lambda, args and kwargs?Middle
Answer
lambda creates a function with one expression, such as key=lambda x: x.score. *args collects positional arguments into a tuple and **kwargs named arguments into a dict; in calls they unpack collections. Prefer def and explicit signatures for complex logic.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How are classes and custom exceptions created and which magic methods help?Middle
Answer
class defines a type; __init__ initializes the instance, __repr__ provides debug representation, __eq__ equality, __iter__ iteration and __enter__/__exit__ the context protocol. class ToolTimeout(Exception): pass creates a framework domain exception. Do not label a timeout a product defect without investigating its cause.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How does import work and how does from import differ?Middle
Answer
import module binds the module name; from module import name binds the selected object. On first import Python locates and executes a module and normally caches it in sys.modules. Side effects, circular imports and binding time matter; mock where the dependency is looked up.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How are generators, decorators and context managers implemented?Middle
Answer
A generator uses yield; a function decorator returns a wrapper and uses functools.wraps; a context manager implements __enter__/__exit__ or @contextmanager with try/finally around yield. Test exception paths, cleanup, preserved metadata and async behaviour. Retry decorators must not repeat side effects without idempotency.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
What are class decorators and closures?Middle
Answer
A class decorator takes a class and returns a modified/replacement class; a function decorator takes a function. This differs from @classmethod. A closure retains enclosing-scope bindings: an evaluator factory may capture a threshold. Watch late binding in loops and shared mutable captured state.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
Where is a dataclass useful in a TAF?Middle
Answer
Dataclasses generate boilerplate for records such as EvalCase or ToolResult. Use default_factory for mutable defaults. frozen=True prevents ordinary field assignment but does not make nested values deeply immutable. Schema validation and invariants need additional code or a dedicated library.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How does a generator differ from a list?Middle
Answer
A generator yields values gradually; a list stores every item. A generator is usually single-use and has no arbitrary index access.
Primary sources
Examples
- A large file can be streamed with a generator without loading every row into memory.
Practice exercises
- Implement batched data reading as a generator and compare memory use with a list.
How do you test exception paths without hiding errors?Middle
Answer
A test should trigger a specific condition, assert the exception type and meaningful attributes, and verify state after failure. A broad except or empty catch hides defects. For retryable failures, separately verify attempt count, backoff, and that non-retryable exceptions are not repeated.
Primary sources
Examples
- pytest.raises checks ValidationError and its invalid_field attribute, while a fixture confirms that no database row was created.
Practice exercises
- Test a function for timeout, validation error, and unexpected error, defining a different policy for each.
What do type hints provide in dynamic Python?Middle
Answer
Type hints document expected contracts and let a static checker catch some errors before runtime. They do not automatically validate values at runtime and do not replace tests. Their greatest value appears at module boundaries, in complex data structures, and during refactoring.
Primary sources
Examples
- A Protocol describes the behaviour required from an API client without coupling code to one concrete class.
Practice exercises
- Add typing to an API-response normalizer and handle Optional and union values explicitly.
03 / 12
AI and ML fundamentals
How do encoders, decoders and Transformers differ and what are embeddings?Middle
Answer
Transformer is an architecture family using attention. Encoders often produce contextual representations; autoregressive decoders generate subsequent tokens; encoder-decoder models combine encoding and generation. An embedding is a vector representation whose similarity may reflect learned relations. It does not guarantee equivalent meaning or factual correctness; evaluate embeddings on domain data.
Primary sources
What ML types exist and what is needed for on-prem LLM hosting?Middle
Answer
Supervised ML learns from labelled targets; unsupervised ML finds structure without those labels; reinforcement learning optimizes behaviour through rewards. On-prem hosting needs licence checks, RAM/VRAM, quantization, serving, concurrency, security and updates. Measure throughput, latency, memory and quality on your tasks.
Primary sources
How do guardrails differ from broader AI safety?Middle
Answer
Guardrails are operational controls in a specific system. AI safety is broader: training choices, alignment, evaluations, misuse analysis, governance and incident response. Do not equate a content filter with overall product safety; test bypasses, false blocks and fail-open behaviour.
Primary sources
04 / 12
Prompting and tool calling
When are few-shot examples and stepwise approaches useful?Middle
Answer
Few-shot prompting supplies representative input/output examples; ensure they do not leak from the evaluation set or introduce bias. Break complex tasks into checkpoints and request concise justification. Do not demand hidden chain-of-thought or use it as an oracle; evaluate artifacts, evidence and actions.
Primary sources
How do temperature and top-k affect output?Middle
Answer
Temperature changes the distribution before sampling; top-k retains k candidates. Greater variability may affect answer stability. Not every API supports these parameters; low temperature or a seed does not guarantee full reproducibility. Record configuration and compare repeated runs on identical cases.
Primary sources
How should code-generation prompts be designed?Middle
Answer
Specify language/runtime, interfaces, dependencies, constraints, acceptance criteria and examples. Request a minimal change, runnable tests and explicit assumptions. Validate generated code through compilation, tests, security checks and review; do not run it with production credentials or unrestricted network/filesystem access.
Primary sources
How does structured output differ from executing a tool call?Middle
Answer
Structured output constrains generated data to a schema but does not prove that values are true. A tool call is the model’s proposal to invoke a capability with arguments; the application controls actual execution. Before a side effect, validate, authorise, ensure idempotency, and require human approval for risky actions.
Primary sources
Examples
- Valid JSON containing deleteUserId still cannot authorise deletion without permission checks and confirmation.
Practice exercises
- Design validation and approval for a tool that changes production configuration.
05 / 12
RAG and retrieval
How are faithfulness and pipeline changes evaluated safely?Middle
Answer
Split answers into claims, inspect support/contradictions, count supported claims and citation coverage for source-requiring claims. Review samples with humans. Version the chunker, embeddings, corpus, index and reranker; changing embeddings usually requires reindexing. Compare A/B variants on identical queries and test TTL/invalidation and rollback.
Primary sources
How do precision and recall apply to retrieval evaluation?Middle
Answer
Recall measures how much of the required material the system found; precision measures how much of the retrieved material is actually relevant. Increasing top-k often raises recall but adds noise and cost. Calculate metrics on labelled queries and add rank-aware measures when result position matters.
Primary sources
Examples
- Search finds all three required documents among twenty results: recall is high, but precision is low.
Practice exercises
- Label relevance for five queries and calculate precision@k and recall@k for two retrievers.
06 / 12
Agents and coordination
What components and types characterize agentic systems?Middle
Answer
Components include perception/context, reasoning/planning, tools/actions, memory, policies and feedback. Classical types are reflex, model-based, goal-based and utility-based; this is a teaching taxonomy rather than a list of modern libraries. Agentic systems select some steps autonomously but need bounded permissions and budgets.
Primary sources
How is agent communication implemented and tested?Middle
Answer
Use typed envelopes, correlation/task IDs, versioning, deadlines, deduplication and explicit ownership. Test duplicate/out-of-order events, missing responses, partial completion and competing writes. Delivery does not imply execution; acknowledgments and business results have different semantics.
Primary sources
When is an agent unnecessary?Middle
Answer
When the action sequence is known and a normal workflow with clear rules and validators can perform it.
Primary sources
Examples
- A fixed parse, validate, then publish pipeline does not need autonomous next-step selection.
Practice exercises
- Convert one deterministic agent scenario into a workflow and compare the risks.
07 / 12
MCP and contract testing
How are schema evolution and optional or extra fields tested?Middle
Answer
Adding an optional field may be compatible; new required fields, narrower types or changed semantics may break compatibility. Extra-field behaviour depends on additionalProperties and reader policy. Missing optional fields differ from null. Test old-client/new-server and the reverse, negotiated protocols and business compatibility.
Primary sources
How are fail-fast contract tests and error handling built in CI?Middle
Answer
On PRs validate schemas and representative examples, diff required fields/types and replay consumer contracts. Separately test protocol errors, tool execution errors, transport failures and provider rate limits. Retry policy belongs to the implementation: test bounded retries, backoff and idempotency. Block breaking contracts before live evals.
Primary sources
08 / 12
Frameworks for chains and graphs
When should AutoGen or CrewAI use one or several agents and what tests apply?Middle
Answer
One agent: simple-task success and controlled failure when a tool is unavailable. Several agents: correct handoff preserving task IDs and denied cross-role actions. Use multi-agent only when coordination/capabilities require it; branding is not justification. Check APIs and support status before implementation.
Primary sources
How are retrieval settings varied and multi-step graphs tested?Middle
Answer
Compare chunk size/overlap, embeddings, top_k and rerankers on versioned golden sets; change one factor or run a controlled experiment. For graphs test input/state schemas, transitions, checkpoints, resume, timeouts and error edges. Assert final state and no repeated side effects; record quality, latency/cost and failures.
Primary sources
How are roles, allowlists, termination and mocks configured?Middle
Answer
Enforce tools at the host/backend with max turns, deadlines, token/cost budgets and cancellation. Stop tokens do not replace workflow termination. Mock adapters with valid, malformed, delayed and error results; isolate time/randomness. Run real-service contract tests separately because mocks may diverge from provider behaviour.
Primary sources
09 / 12
GenAI test automation
Which semantic metrics can be automated?Middle
Answer
Embedding cosine similarity, token overlap and model-based rubrics can help but measure different things. Exact match suits IDs/enums, token overlap misses some semantics and cosine does not prove truth. Combine domain validators, factual checks and human calibration; explain metric limitations in reports.
Primary sources
What must be versioned to reproduce an LLM-system result?Middle
Answer
Record the model and parameters, system and developer prompts, tool schemas, application code, retrieval corpus, chunker, embedding model, index, reranker, and evaluation dataset. A provider model alias may change without a code commit, so run metadata should keep the available concrete identifier and timestamp.
Primary sources
Examples
- The same prompt changes after an index update; the trace reveals different retrieved chunks and corpus versions.
Practice exercises
- Create a run manifest sufficient to compare two releases and explain a quality-metric change.
10 / 12
Evaluation and golden datasets
How is RAG evaluated and which evaluation frameworks exist?Middle
Answer
Separate retrieval, generation and whole-system outcomes. Examples include Ragas, DeepEval, OpenAI Evals/API evals, LangSmith and Langfuse. They differ in datasets, judges, tracing and metrics; they are not interchangeable guarantees. Choose by required assertions, privacy, reproducibility and raw-result export. As of 2026-10-06, OpenAI is moving users away from the deprecated Evals platform toward Datasets/newer evaluation workflows; check the current deprecations page before integration.
Primary sources
Why is LLM-as-judge not an absolute oracle?Middle
Answer
The judge can also be wrong and biased; it needs calibration against human ratings and deterministic checks.
Primary sources
Examples
- A judge may consistently prefer longer answers regardless of accuracy.
Practice exercises
- Compare judge scores with human labels and identify systematic disagreements.
11 / 12
GenAI security and privacy
What are major threats and what is jailbreaking?Middle
Answer
Threats include prompt injection, sensitive disclosure, poisoning, unsafe output handling, excessive agency and resource abuse. Jailbreaking attempts to bypass model policies; injection redirects behaviour through untrusted instructions. Mitigate with scoped tools, validation, isolation, access control and monitoring. Filters or system prompts alone are insufficient.
Primary sources
12 / 12
Responsible AI
What ethical principles and privacy challenges matter?Middle
Answer
Translate fairness, transparency, privacy, accountability, safety and human oversight into observable requirements. Risks include unjustified reuse, re-identification, sensitive inference and unequal group quality. Agree on intended use, affected users, appeal/escalation and accountable owners; abstract principles alone are insufficient as test oracles.
Primary sources
How do anonymization and pseudonymization differ?Middle
Answer
Anonymization aims to make re-identification impractical under the relevant threat model; pseudonymization replaces identifiers but permits linkage through additional information. Removing names alone does not ensure anonymity. Check indirect identifiers, rare combinations and logs; consider privacy–utility trade-offs and access to mapping keys.