← Back to Knowledge Base

KNOWLEDGE / 13

AI System Testing

A bilingual learning path across 12 topics, from test process and Python to RAG, MCP, agents, evaluation, security, and Responsible AI.

Reviewed: 2026-10-06

Testing is the static and dynamic examination of a product and related work products to identify defects and determine alignment with requirements and user needs. This is a teaching definition, not a verbatim standards quotation.

Check deterministic properties with code assertions and semantic quality with a defined rubric, calibrated evaluation, and human review. Diagnose model, retrieval, application, tool, and environment failures separately.

Work through one topic at a time: answer from memory, read the explanation, complete the exercise, and preserve evidence. Numeric thresholds and sample sizes are illustrative rather than normative.

Learning project

Use a read-only AI Quality Architect over fictitious Jira/Confluence fixtures. It analyses requirements and coverage and creates a cited strategy report. Add write tools only after separate permission and approval validation so datasets, retrieval, reasoning, and execution can be evaluated independently.

Completion criteria

Junior
Create 10 cases with inputs, contexts, expected properties, and forbidden actions; implement schema and exact-ID checks.
Middle
Add retrieval metrics, mocked tool failures, a two-turn conversation, and paired prompt comparisons. Preserve versioned traces and sanitized evidence.
Senior
Justify dataset coverage, judge calibration, release gates, residual risks, cost, and privacy. Demonstrate recovery and rollback and explain which claims the evidence does not establish.

Topic completion means an explanation in your own words, a runnable example, a verified negative case, a report with denominators, versions and evidence, explained limitations, and equivalent Ukrainian/English content. A high quiz score alone does not establish Senior competence.

01 / 12

Documentation and testing process

02 / 12

Python for automation

03 / 12

AI and ML fundamentals

04 / 12

Prompting and tool calling

05 / 12

RAG and retrieval

06 / 12

Agents and coordination

07 / 12

MCP and contract testing

08 / 12

Frameworks for chains and graphs

09 / 12

GenAI test automation

10 / 12

Evaluation and golden datasets

11 / 12

GenAI security and privacy

12 / 12

Responsible AI