CCA Professional · M2 — Enterprise Integration & Production

Exam domains covered: D3 — Integration (19%), D4 — Evaluation, Testing & Optimization (16%)

Approximate module runtime: 158 minutes of Anthropic learning content

The Evaluation Grading Ladder

Use the cheapest reliable method for each task. Only escalate when the tier below cannot be trusted.

1. Code-based evaluation
       ↓ (when interpretation is required)
2. Model-based evaluation (LLM as judge)
       ↓ (when neither above can be trusted)
3. Human review

Code-Based Evaluation

Use when the output can be checked mechanically:

• Correct JSON schema
• Correct field extraction (name, date, amount)
• Correct arithmetic
• Exact format compliance (correct heading names, section order)
• Presence/absence of required elements

No model involvement needed. Fastest and cheapest.

Model-Based Evaluation (LLM as Judge)

Use when the output requires interpretation:

• Tone and brand alignment
• Reasoning quality
• Argument coherence
• Safety and policy compliance at scale

Three required design elements for a trustworthy LLM judge:

1. Different model from the evaluated model — self-grading produces rationalisation, not evaluation
2. Constrained verdicts — "pass/fail" or "acceptable/unacceptable/escalate", never "score 1–10 with a paragraph explanation" (the judge's uncertainty gets absorbed into the score, not surfaced)
3. Calibrated against human-labeled outputs before use at scale — run the judge against a set of human-graded samples and confirm agreement before trusting it

The uncalibrated judge rule: An uncalibrated LLM judge is worse than no automated grade. It produces a confidence signal that looks like measurement while silently disagreeing with what humans would actually score. This is one of the most-tested traps in Domain 4.

Human Review

Reserve for:

• Cases where neither code-based nor model-based can be trusted
• Genuinely ambiguous outputs where interpretation requires domain expertise
• Calibration exercises (grading samples the LLM judge will later use)
• Regulated decisions requiring a named human decision-maker

Golden Dataset Design

The evaluation dataset must be:

What makes a bad golden dataset:

• Synthetic questions generated by a model from documentation → inherits the documentation's blind spots
• Examples already used in the system prompt → only proves memorisation, not generalization
• Questions written by the engineers who built the system → tests the builders' assumptions with the builders' assumptions

The POC-to-Production Gap

A proof of concept reliably hides four dimensions:

The exam tests this with questions describing a "successful demo." The architect's job is to name which of these four the demo didn't test, and what evidence is needed before production commitment.

Reliability Controls — Which Layer

This is one of the most-tested mechanical facts in Domain 3. Place each control at exactly its layer:

The placement rule: Each control must be at its layer. A retry at the orchestration layer retries an entire multi-step workflow, compounding cost and side-effects. A circuit breaker at the API call layer won't help when the failure is at the service boundary.

A/B Testing Rigor

Four elements that must exist before an A/B test starts:

1. Pre-committed hypothesis — what you expect to change and why. Prevents p-hacking (looking at results, then deciding what to "measure")
2. Power calculation — minimum sample size for the effect size you care about. A test that runs for one week without a power calculation may be underpowered and produce meaningless results
3. One primary metric — with secondary guardrail metrics. Not "we'll see what moves"
4. Statistical significance gate — commit to the threshold before looking at results, not after

An offline evaluation suite gain (e.g., +6 points) doesn't guarantee a production gain. After an offline win, the next step is a controlled A/B test on a fraction of real traffic with quality and guardrail monitoring — not full deployment.

Shadow Testing vs Canary Rollout

Shadow testing trades real downstream signal for zero exposure. It's often the only acceptable path in regulated deployments. A canary gives you real signal but requires accepting that some users see the new behavior.

Change Attribution

When a production metric moves, three causes must be distinguished before any fix is applied:

Mixing these up produces the wrong fix:

• Attributing data drift to model drift → rollback that doesn't help
• Attributing a model-update effect to data drift → expensive corpus work that doesn't help

Leading vs Lagging Indicators

The exam tests whether you know which signals are early warning vs. post-failure:

The question pattern: "which signal gives the earliest warning of quality degradation?" → always the pipeline-internal metric.

Observability Strategy at Scale

At tens of thousands of sessions per day, full payload capture on every session is costly and still makes failures hard to find. The correct strategy:

• Structured traces with correlation IDs on every session — lightweight, fast to query
• Key metrics (latency, retrieval relevance, tool error rates) on every session
• Full payload capture (complete prompts and responses) on a sample, plus triggered on error conditions

This way: failures are always findable (every failure triggers full capture), and routine operational cost stays low (not logging 200-token prompts and responses for every session at full volume).

Integration Patterns

MCP vs Direct API vs Agent-to-Agent

MCP earns its complexity when: many systems, many consuming applications, decentralised ownership, and frequent tool churn. Each team maintains its server; every application reuses it without re-implementing the connection.

Tool Bloat and Progressive Discovery

Symptoms of tool bloat:

• Tool-selection accuracy declining as new tools were added
• A third of registered tools never invoked in production
• Latency increasing with each added tool

Fixes, in order:

1. Audit and remove unused or overlapping tools
2. Progressive discovery — only load the tools relevant to each request, not the full catalogue
3. Rewrite tool descriptions to be explicit, non-overlapping, and boundary-clear

RAG Chunking Strategy

Fixed-size chunking (e.g. 300 tokens) severs clauses from the definitions and cross-references that give them meaning. The fix for document-heavy RAG over structured documents (contracts, regulations, standards):

• Structure-aware chunking — aligned to clauses and sections, not character count
• Metadata linking — chunk references its parent section, definitions, and cross-references
• Result: retrieval returns interpretable fragments, not isolated text islands

Hybrid retrieval — for corpora with both natural-language queries and exact identifiers (part numbers, IDs, codes):

• Exact identifiers → lexical/keyword matching wins
• Natural-language queries → semantic embedding wins
• Solution: hybrid retrieval combining both, weighted by query type detection

Cost Optimization — Order of Operations

Never apply cost levers blind. The disciplined approach:

1. Trace first — analyse token usage and cost per workload from production traces
2. Identify the dominant driver — is it context size, caching misses, output length, or model tier?
3. Apply targeted levers:
   • Large static prefix → prompt caching
   • Inflated context → context trimming (remove padding, limit retrieved chunks)
   • High output length → output length controls
   • All levers exhausted → evaluate model downsizing with quality measurement
4. Evaluate quality at each step — assume nothing, measure everything

"Switch every workload to the smallest model" applied blindly is never the right first step. First understand where the spend is.

System-Prompt Architecture — Four-Component Structure

A well-built system prompt has exactly four components, in this order:

1. Role definition — who is the model and what is its authority?
2. Testable criteria — what specific behaviors should it produce?
3. Boundary statements — what must it never do, and what should it say when asked?
4. Output format — how should results be expressed?

A criterion is testable if you can write an eval test case that passes or fails based on whether the model meets it. Underspecification is a gap the model fills with its own assumption — differently each time.

Boundary statement structure has two parts: explicit prohibition (name the specific action/topic out of scope) and refusal response (specify exactly what the model should say when triggered).

A fake or underspecified guardrail is worse than no guardrail — it creates false confidence. Templates separate "what changes" (slots) from "what must never change" (scaffolding), so filling a slot can't accidentally remove a constraint.

Few-shot examples: use 2–4 worked examples, not 10+. Always include a rejection example — a case that looks like it should trigger the behavior but should not. Examples showing only one kind of case silently bias the model toward that case.

Context Strategy Spectrum

Four strategies for managing context across a session. Production systems almost always combine them.

Progressive is the right default for most production workloads; monolithic is the exception for genuinely bounded tasks. Context strategy (how it reaches the model) and context sizing (how much fits) are two separate decisions.

Trimming examples:

• Tool returns 6,000-token JSON; next step needs 4 fields → extract the 4 fields (~80 tokens), discard the rest
• Conversation history reaches 18,000 tokens → replace with a 500-token running summary + last 3 turns verbatim
• Code file from an earlier step no longer referenced → remove it from context

Four context terms not to conflate:

Failure Modes by Architecture Type

Four Logging Layers

Capture all four for every session — not just the final response:

Every tool invocation must be attributable. Four things, every time: who triggered it (user identity), with which permissions (access rights used), against which resource (what was accessed), and when (timestamp). Logs that only record a service-account identity provide no attribution — this is a security gap the exam asks you to identify directly.

Credential Storage — What's Allowed

Accuracy-Latency Enhancement Costs

Any enhancement that improves accuracy adds latency. A correct answer names the quality benefit, states the latency cost, and verifies the latency still fits the SLA.

The discipline here is to run the numbers: if p95 + the added latency still lands under the SLA, adopt and monitor. Don't reject accuracy improvements on principle.

Module 2 Trap Patterns

Trap 6 — Aggregate metric hides tail: Average accuracy of 94% looks good while a specific document type, language, or query pattern fails at 60%. Always check for tail populations before declaring a system healthy.

Uncalibrated judge (Trap variant of Trap 5): A judge that correlates 0.6 with human raters is worse than no automated grade — it produces a false sense of coverage. Calibrate before trusting at scale.

Key Rules to Memorise

• Eval grading ladder: code → model (judge) → human. Use cheapest reliable method
• LLM judge must use a different model, constrained verdicts, and be calibrated against human labels
• Reliability controls: retry (API layer) / circuit breaker (service boundary) / fallback (orchestration)
• A/B test requires: pre-committed hypothesis + power calculation + one primary metric + significance gate
• Shadow testing = logged but never shown. Canary = real users see it
• Change attribution: model drift vs data drift vs model-update effect — each needs a different fix
• Leading indicators: retrieval relevance and grounding rates move before user-visible quality
• MCP: many systems, many apps, decentralised ownership. Direct API: single pipeline, owned
• System prompt: role definition → testable criteria → boundary statements → output format
• Testable criterion: you can write an eval test case that passes or fails based on it
• Context strategies: monolithic / progressive / retrieval / compaction — production usually combines them
• Retrieval augments context; it does NOT expand the context window
• The model has zero native memory — your application stored and passed it back
• Agent breaks first on unbounded tool use; RAG on retrieval quality drift
• Log all four layers: request / response / context / outcome
• Tool attribution: who + which permissions + against which resource + when
• Credential storage: env vars or secrets manager only — never source code, prompts, or README
• Accuracy improvements add latency — always verify the SLA still holds after adding the enhancement