CYBRARY Northstar Benefits demo suite
Third-party benefits administration. One incident — OAUTH-1181 — unreviewed enterprise consent grant on the claims tenant — worked eight times. Only the model runtime changes, so every difference in the room is attributable to the runtime decision rather than to a different story.
8 stages · 9 hours end to end · tenant northstar-claims.onmicrosoft.example
- Priya Raman — SOC lead, runs the bridge
- Dan Whitlock — incident responder, owns containment
- Ana Sousa — AI platform engineer, owns runtime configuration
- Marcus Bell — governance lead, owns what may be recorded and shown
Work OAUTH-1181 with nothing but the console, the sign-in logs and the consent record.
Establish the human baseline: time to triage, and what the analyst actually had to read.
Debrief: What took the longest, and was any of it judgement rather than lookup?
40 min · SOC analyst, Incident responder
Same case, with an on-device assistant that never sends a token off the laptop.
Privacy comes free; capability does not. The small local model summarizes but misreads scope.
Debrief: Which of R1's slow steps did the local model actually remove?
40 min · SOC analyst
The analyst supplies their own provider key and picks a model mid-case.
A key in a browser is a control problem before it is a capability win.
Debrief: Who owns the prompt, the key and the log in this configuration?
55 min · SOC analyst, AI consumer
Northstar's regulated gateway is wired in: their endpoint, their retention, their allow-list.
The gateway makes the assistant auditable — and makes every limitation the company's own.
Debrief: Which controls moved from the lab to the enterprise, and who is accountable now?
75 min · AI platform administrator, Security architect, SOC lead · shared session
Two candidates work the same case with different assistants; the grade must stay fair.
Ground truth is the scenario predicate, never the model's opinion.
Debrief: What would have made this assessment indefensible?
60 min · Incident responder, Assessment designer · shared session
A provider degrades and the allowance crosses 80% mid-incident.
Routing and budget policy are incident controls, not procurement details.
Debrief: Did the fallback preserve the investigation, or just preserve the uptime number?
80 min · AI operations engineer, Security architect · shared session
The whole team works one run: one control state, one trace, one allowance, one join code.
Shared AI needs shared discipline — who may move a control, and who may quote the model.
Debrief: Where did the budget actually go, and what did the team learn from it?
120 min · SOC lead, Incident responder, Threat hunter, Communications lead · shared session
Same prompts, same context budget, same tools — four candidate models, one scorecard.
A comparison without a held-constant harness is theater.
Debrief: Which model would you actually put in front of Northstar's analysts, and why?
90 min · AI security lead, Governance lead, SOC leadership · shared session