No model has produced a P-report JSON for the five gold diagrams. The prompt-series score is what Track A asks for. The schema score is what a complete fill can hold. The last two rows are files already in the repository.
Scores
Each check is 0 to 4. Twelve checks, 48 points. 4 matches the gold report. 3 is present and usable. 2 is thin. 1 is hinted. 0 is absent. Letter cut: A from 42, A- from 36, B from 29, C from 18, D from 8.
| Artifact | Points | Grade | What was scored |
|---|---|---|---|
| Prompt series v1.1 | 47 / 48 | A | What Track A asks the model to write. |
| Specified JSON | 44 / 48 | A | What schema.json can hold. Reviewer is a human field, so this column cannot reach 48. |
| Gold rag-app inventory | 8 / 48 | D | Q1 inventory only. Auspex would not use a manual threat list as gold; these files follow that limit. |
| Zeroshot stub (rag-app) | 9 / 48 | D | Intentional miss for the scoring scripts. T1 is a catalog line, referent missing, no control point. |
The three gold reports
- Trail of Bits Kubernetes assessment (2019). Commissioned by the Kubernetes Security Working Group. Twelve person-weeks on v1.13.4. A threat model and data-flow analysis of prioritized components, then findings that test namespace and Pod isolation claims. sig-security 2019 findings.
- RFC 6819, OAuth 2.0 Threat Model. IETF consensus. Scope and attack assumptions first. Section 3 lists features that already exist. Section 4 groups threats by client, authorization endpoint, token endpoint, and resource access. Each threat has a mechanism and countermeasures. RFC 6819.
- CNCF Financial User Group Kubernetes threat model. STRIDE on trust boundaries, then attack trees to DoS, malicious code, and persistence, plus two scenarios (compromised application, attacker on the network). SDLC and monitoring apps are out of scope. Each leaf names a mitigation and a validation snippet. k8s-threat-model.
RFC 6819 states attack assumptions and existing features before any threat. Trail of Bits pins a version and tests trust-boundary claims. CNCF writes a mitigation and a validation snippet on each leaf. All three are documents a reviewer can read without a parser.
Per-check scores
Columns: prompts (what the chain asks), schema (what the JSON can hold), gold inventory, zeroshot stub.
| Check | Prompts | Schema | Gold inv. | Stub | How this pack treats it |
|---|---|---|---|---|---|
| Named system, version, and scope | 4 | 3 | 2 | 2 | P-norm asks for source_id, version, and commit, or unknown. Gold inventory has no version pin. |
| Attacker capabilities stated | 4 | 4 | 1 | 1 | P-adv writes assumptions and positions per zone. Threats must cite attacker_position. |
| Architecture and trust boundaries | 4 | 4 | 4 | 3 | P-norm, P-diag, and P-sol. Gold rag-app has four zones. Stub drops model-api and llm_subset. |
| In-scope and out-of-scope named | 4 | 3 | 1 | 0 | P-scope. Schema stores the arrays. Gold inventory has no scope lists. |
| Existing security features listed first | 4 | 4 | 0 | 0 | P-controls lists only features shown on the diagram. Empty is allowed only when none_drawn is true. |
| Each threat names a component or flow | 4 | 4 | 0 | 0 | diagram_referent must exist in inventory. Stub T1 uses referent missing. |
| Preconditions or attacker position | 4 | 3 | 0 | 0 | P-stride, P-phantom, and P-dedup require attacker_position from P-adv. Schema field is optional so old fixtures still validate. |
| Action at a named control point | 4 | 4 | 0 | 1 | P-act requires a control_point. Mitigate and eliminate also require validation (test, log, or fail_condition). |
| What the design does not claim to stop | 4 | 4 | 0 | 1 | P-adv writes claim_boundary.does_not_claim and box. P-report repeats that list. |
| Authors, method, and review | 3 | 3 | 0 | 1 | P-report sets method and date. Reviewer stays empty for a human. Filling it in the prompt would be a fake sign-off. |
| A reviewer can read it without a parser | 4 | 4 | 0 | 0 | P-report writes report.markdown. Pack v1.3 adds P-export-md, P-export-json, and P-export-csv after the tracks. |
| A way to check a mitigation landed | 4 | 4 | 0 | 0 | Mitigate and eliminate require action.validation. P-qa checks actions_have_validation. |
Verdict on the prompt series
Track A v1.1 adds P-adv (attacker capabilities and claim boundary), P-controls (features already on the diagram), action.validation on mitigate and eliminate, and P-report (readable markdown). Inventory, STRIDE, PHANTOM-B, and control_point actions were already at 4 of 4. A complete P-report fill meets this rubric at A. Attack trees stay optional. DREAD is out. There is still no gold threat list.
| Step | Taken from | What it writes |
|---|---|---|
| P-adv | RFC 6819 section 2.2; CNCF scenarios | Assumptions, positions per zone, and claim_boundary. |
| P-controls | RFC 6819 section 3; Trail of Bits isolation discussion | Authn, TLS, filters, or isolation already drawn, or none_drawn. |
| P-act validation | CNCF leaf tests; Trail of Bits retestable findings | test, log, or fail_condition on the named control_point. |
| P-report | All three gold reports | report.markdown plus method and date. Reviewer left empty. v1.3: P-export-md / P-export-json / P-export-csv write the files. |
Machine scores in the repository
Inventory precision and recall, format Jaccard, PHANTOM-B and STRIDE
coverage, and schema checks run from
eval/threat-model/ without calling a model. Those scripts
do not award the letter grades on this page. SME sheets in
sme/ are still empty, so closure stays false.
Schema:
eval/threat-model/schema.json.
Prompts:
/tools/prompts/threat-model/.
Vendor demo PDFs, fictional STRIDE samples, and Shostack's 2020 PCI reverse-engineering draft were not used. The PCI draft analyzes a standard, not a deployed system.