AI SYSTEM TESTING
Junior
Foundations and basic checks across the twelve AI testing topics.
01 / 12
Documentation and testing process
What is the test documentation pyramid?Junior
Answer
It is a useful hierarchy: policy and strategy → plan → scenarios and cases → data and procedures → execution evidence and reports. It is neither a universal standard nor the automation test pyramid. Add an evaluation rubric, dataset card and model versions for AI.
Primary sources
How does a TMS differ from a bug tracker?Junior
Answer
A TMS manages test assets, suites, runs, coverage and results. A bug tracker manages defects, triage and the fixing workflow. Integration provides case → run → defect → fix traceability; product capabilities can overlap.
Primary sources
What are a test basis and a test oracle?Junior
Answer
A test basis is the set of sources from which we derive test conditions: requirements, contracts, models, risks, laws, or the observed behaviour of an earlier version. A test oracle is the means used to determine the expected result. It may come from a specification, business rule, reference system, calculation, or expert judgement, and its reliability must also be assessed.
- For an open-ended AI task, the oracle is often a set of constraints, supported facts, acceptable outcomes, and a rubric rather than one exact string.
Primary sources
Examples
- For a discount calculation, the loyalty-program rules form the test basis, while an independent calculation can serve as the oracle.
Practice exercises
- For a familiar feature, list the available test basis, choose an oracle, and identify how that oracle could be wrong.
What are test types, and how do they differ from test levels?Junior
Answer
A test type defines the objective of a check or the quality characteristic being investigated. Unlike a level, a type is not tied to one scale of the system and can be applied at several levels:
- Functional testing checks what the system does: rules, calculations, transitions, and outcomes.
- Non-functional testing checks how the system operates: performance, reliability, security, accessibility, compatibility, and other characteristics.
- White-box, or structural, testing uses the system’s internal structure: statements, branches, paths, or data flows.
- Change-related testing includes confirmation testing of corrected behaviour and regression testing for unintended side effects.
- For AI, functional checks cover responses, retrieval, tool use, permissions, and fallback; performance, security, fairness, and robustness remain separate quality aspects.
Primary sources
Examples
- System is the level at which the complete product is checked. Functional, performance, security, and regression testing are different test types that may all be applied at that level.
Practice exercises
- For file upload checks, identify the level of each check and classify its type as functional, non-functional, structural, or change-related.
02 / 12
Python for automation
What do mutable and immutable mean?Junior
Answer
A mutable object changes in place: list, dict, set. An immutable object does not: int, str, tuple. Assignment binds a name to an object; it does not copy it. A tuple may contain a mutable list, so container immutability does not mean deep immutability.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How do list, set and dict differ?Junior
Answer
A list is an ordered sequence allowing duplicates; a set holds unique hashable elements without guaranteed order; a dict maps hashable keys to values and preserves insertion order. Use sets for membership, lists for indexed sequences and dicts for lookup by ID; consider memory and order.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
What are comprehensions and are they always better than loops?Junior
Answer
[i*i for i in range(3)] builds a list; {i: i*i for i in range(3)} builds a dict. They are concise for simple transformations, but not automatically better: complex logic, side effects and large outputs may favour loops or generators. Measure performance on actual data.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How do range and slicing work?Junior
Answer
range(start, stop, step) represents an integer sequence without creating a full list; stop is excluded and step cannot be 0. seq[start:stop:step] selects a slice; negative indices count from the end. A list slice is a shallow copy; other libraries may use different view/copy semantics.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
How are objects copied and JSON serialized?Junior
Answer
copy.copy creates a shallow copy sharing nested objects; deepcopy copies recursively subject to type behaviour and custom hooks. json.dumps/loads use strings; dump/load use file objects. Datetime, set and custom classes need explicit conversion; serialization does not replace schema validation.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
What is self and which package managers exist?Junior
Answer
self is the conventional name of the first instance-method parameter giving access to the instance. pip installs Python packages; Poetry manages project dependencies/packaging; conda manages environments and packages including native dependencies. Use one coherent dependency workflow and pin versions.
Primary sources
- S01: Python Tutorial (Python 3.14)
- S02: Python Data Structures (Python 3.14)
- S03: Python Control Flow (Python 3.14)
- S04: Python Errors and Exceptions (Python 3.14)
- S05: Python Modules (Python 3.14)
- S06: Python typing (Python 3.14)
- S07: Python dataclasses (Python 3.14)
- S08: Python contextlib (Python 3.14)
- S09: Python copy (Python 3.14)
- S20: Python functools (Python 3.14)
Why use a Python context manager?Junior
Answer
A context manager defines entry into a controlled context and guaranteed exit, even when an exception occurs inside it. It suits files, locks, transactions, temporary resources, and test fixtures because setup and cleanup stay together.
Primary sources
Examples
- with open(...) closes a file automatically; a custom context manager can roll back a test transaction.
Practice exercises
- Create a context manager for a temporary environment variable that always restores the previous value.
03 / 12
AI and ML fundamentals
How do AI, ML and Deep Learning differ?Junior
Answer
AI broadly concerns systems performing tasks requiring intelligent behaviour. ML learns patterns from data rather than relying only on explicit rules. Deep Learning is ML using multilayer neural networks. A rule-based agent may also be AI; not every AI system is an LLM.
Primary sources
How do completion models and assistants differ and which families matter?Junior
Answer
Completion describes continuation generation; chat/instruction interfaces introduce roles and task following. Assistant may mean an application with tools and memory rather than a distinct architecture. Examples include OpenAI, Gemini, Claude, Llama, Mistral, DeepSeek and Phi. Select through evals, hosting, licence, privacy and cost rather than branding.
Primary sources
What are AI guardrails?Junior
Answer
Guardrails are concrete input, output and action constraints/checks: schema, policy, permissions, budgets and moderation. They reduce specified risks but can have false positives/negatives. The application should enforce denial of unsafe actions rather than relying only on model instructions.
Primary sources
What are tokens and a context window in an LLM system?Junior
Answer
A model processes text as tokens, while the context window limits the combined size of instructions, history, retrieved data, and output. More context does not guarantee a better answer: irrelevant text increases cost, latency, and the chance of losing important signals. Select and structure context deliberately.
Primary sources
Examples
- A long log dump can obscure acceptance criteria even when everything technically fits in the window.
Practice exercises
- Allocate a prompt budget among instructions, task data, retrieved evidence, and output, explaining priorities.
04 / 12
Prompting and tool calling
What are system and user messages for?Junior
Answer
System messages provide application instructions; user messages contain the current task and input. Exact roles and precedence depend on the API. Treat retrieved pages and tool outputs as data, not permission to change policy. The backend must enforce permissions independently of prompt text.
Primary sources
How do simple and structured prompts differ?Junior
Answer
A simple prompt is a short task. A structured prompt separates objective, context, constraints, output format and examples. Structure helps consistency but does not guarantee correctness. Define insufficient-evidence behaviour, bound scope and test template variables; keep secrets out of prompts.
Primary sources
05 / 12
RAG and retrieval
How are grounding, truncation, caches and stale sources checked?Junior
Answer
Check grounding as claim → supporting passage rather than URL existence alone. Add facts at the start/end of long contexts, changed sources and invalidated caches. Check document/version IDs, timestamps, retrieval access and user-scoped cache keys. Grounded answers may repeat false sources, so verify factual truth separately.
Primary sources
What does RAG add to an LLM?Junior
Answer
RAG retrieves relevant external material and adds it to the generation context; it does not guarantee a correct or complete answer.
Primary sources
Examples
- An assistant may answer from a current specification and still interpret it incorrectly.
Practice exercises
- Compare answers with correct, missing, and conflicting retrieval context.
06 / 12
Agents and coordination
What is an AI agent and the observe think act reflect loop?Junior
Answer
An agent selects actions toward a goal using observations. Observe reads state; think/plan selects the next step; act calls a tool; reflect evaluates the result. This is a conceptual model, not a requirement to expose private reasoning. Test observable actions, outcomes and stop conditions.
Primary sources
What memory should be persisted?Junior
Answer
Short-term state contains the current task and intermediate results; episodic memory records past runs; semantic memory holds facts/knowledge, sometimes in a vector index. Persist only what is needed with provenance, scope, TTL and deletion. Avoid secrets or unbounded raw scratchpads; test cross-user isolation and outdated memory.
Primary sources
Which agent communication patterns exist?Junior
Answer
Direct messaging addresses a particular agent; a blackboard shares state; request/response awaits a reply; pub/sub distributes events. Define sender, task/run IDs, schema, delivery semantics and timeouts. Shared state must not allow cross-task reads or evidence overwrites.
Primary sources
07 / 12
MCP and contract testing
What is MCP and why does it matter to QA?Junior
Answer
Model Context Protocol standardizes host/client connections to servers exposing tools, resources and prompts. QA tests the protocol, schemas, permissions and observable business effects. MCP is not an agent framework and does not make tools safe by itself. Protocol versions and negotiated capabilities are part of the test environment. This material pins MCP revision 2026-07-28; older handshake-based revisions have a different lifecycle, so interoperability tests must state the protocol version explicitly.
Primary sources
How are schemas, responses and basic negative cases checked?Junior
Answer
Validate inputSchema using JSON Schema; handle missing required fields, wrong types, bounds and unknown tools predictably. Check result envelopes and content blocks. When a tool provides outputSchema, validate structuredContent; do not require an arbitrary output schema from every tool. A successful response does not establish business correctness.
Primary sources
What risks occur in MCP tool calls?Junior
Answer
Risks include wrong arguments, unauthorized access, injected tool/resource text, sensitive outputs, unintended writes and duplicate side effects. Hosts and servers should validate and authorize independently. Do not treat tool descriptions/annotations as trusted authority. Use fixtures instead of real secrets and test permission boundaries.
Primary sources
08 / 12
Frameworks for chains and graphs
Which minimal pipeline should be tested in LangChain or LlamaIndex?Junior
Answer
Input → prompt template → optional retriever → model adapter → parser/validator. Automate missing-template-variable and invalid-structured-output cases; also verify preserved source IDs. LangChain/LlamaIndex are implementation choices; frameworks do not replace an oracle or test strategy.
Primary sources
09 / 12
GenAI test automation
What is similarity search used for in GenAI QA?Junior
Answer
It finds nearby vector representations for retrieval, deduplication and semantic comparison. It is a tool, not an oracle: allowed and not allowed may be close. Use targeted assertions for numbers, negation, entities and permissions; calibrate thresholds on labelled pairs.
Primary sources
Which LLM-system checks can be deterministic?Junior
Answer
Schema validity, required fields, citation existence, permission filters, tool arguments, latency, and token limits can be checked with ordinary code. Semantic correctness often needs a reference, rubric, human review, or calibrated judge. Use a cheap, unambiguous oracle before delegating everything to another model.
Primary sources
Examples
- A validator rejects an answer containing a citation ID that is absent from retrieved documents.
Practice exercises
- Classify ten quality criteria as deterministic, model-based, or human-reviewed checks.
10 / 12
Evaluation and golden datasets
What makes an evaluation dataset useful?Junior
Answer
A dataset should represent real tasks, risky edge cases, negative examples, and expected evidence. Labels need definitions and a review process. Random attractive prompts create a demo rather than a baseline; also protect the dataset from leakage into prompts or the retrieval index.
Primary sources
Examples
- A requirement-review set includes complete, vague, contradictory, and intentionally unresolvable requirements.
Practice exercises
- Create ten evaluation cases with a risk category, expected behaviour, and scoring criteria.