KNOWLEDGE / 13
AI System Testing
A bilingual learning path across 12 topics, from test process and Python to RAG, MCP, agents, evaluation, security, and Responsible AI.
Reviewed: 2026-10-06
Testing is the static and dynamic examination of a product and related work products to identify defects and determine alignment with requirements and user needs. This is a teaching definition, not a verbatim standards quotation.
Check deterministic properties with code assertions and semantic quality with a defined rubric, calibrated evaluation, and human review. Diagnose model, retrieval, application, tool, and environment failures separately.
Work through one topic at a time: answer from memory, read the explanation, complete the exercise, and preserve evidence. Numeric thresholds and sample sizes are illustrative rather than normative.
Learning project
Use a read-only AI Quality Architect over fictitious Jira/Confluence fixtures. It analyses requirements and coverage and creates a cited strategy report. Add write tools only after separate permission and approval validation so datasets, retrieval, reasoning, and execution can be evaluated independently.
Completion criteria
- Junior
- Create 10 cases with inputs, contexts, expected properties, and forbidden actions; implement schema and exact-ID checks.
- Middle
- Add retrieval metrics, mocked tool failures, a two-turn conversation, and paired prompt comparisons. Preserve versioned traces and sanitized evidence.
- Senior
- Justify dataset coverage, judge calibration, release gates, residual risks, cost, and privacy. Demonstrate recovery and rollback and explain which claims the evidence does not establish.
Topic completion means an explanation in your own words, a runnable example, a verified negative case, a report with denominators, versions and evidence, explained limitations, and equivalent Ukrainian/English content. A high quiz score alone does not establish Senior competence.
01 / 12
Documentation and testing process
02 / 12
Python for automation
03 / 12
AI and ML fundamentals
04 / 12
Prompting and tool calling
05 / 12
RAG and retrieval
06 / 12
Agents and coordination
07 / 12
MCP and contract testing
08 / 12
Frameworks for chains and graphs
09 / 12
GenAI test automation
10 / 12
Evaluation and golden datasets
11 / 12
GenAI security and privacy
12 / 12