AI security curriculum

What an AI system is made of once you stop calling it 'the model': prompts, context, identity, tools and the trust boundaries between them — and what goes wrong in the application layer that wraps it.

• Inventory a live AI application: model, prompt, corpus, identity and tool tokens.
• Read a trace and say which component made a decision and on whose authority.
• Treat a guardrail as a control with a regression test, not a one-off prompt.

Where the untrusted input actually enters: a document in the corpus, a tenant's data in a shared index, a tool result the agent believes, and a memory that carries an injection into tomorrow's session.

• Trace an indirect prompt injection from retrieved content to the action it caused.
• Prove tenant isolation in a retrieval index rather than assuming the filter holds.
• Scope agent tool permissions and require approval where an action is irreversible.

Attacks on the model itself — extraction, membership inference, evasion — and how a campaign is run and reported against an AI system without turning it into anecdotes.

• Recognize extraction, inference and evasion from what the telemetry looks like.
• Choose a defense that fits the attack rather than the headline.
• Run and report an AI red-team campaign with reproducible evidence.

Using AI on the defensive side without trusting it blindly: hallucinated indicators, an autonomous responder that needs a leash, incident command when the AI system is the incident, and measuring an agent's trajectory rather than its confidence.

• Verify an AI-produced indicator before it reaches a block list.
• Bound an autonomous responder with scope, approval and a rollback path.
• Judge an agent on evidence and trajectory, not on how sure it sounded.