defense / Incident response
Product security incident response
Validate a vulnerability report, scope affected products, contain active exploitation, ship fixes, and publish an accurate advisory.
- Objective
- Turn an inbound vulnerability report into a verified scope, working containment, tested fixes, and accurate customer guidance before exploitation spreads.
- Starting access
- Report queue, repositories, CI, product telemetry, deployment inventory, release, and advisory tools
Model results / v0.1.0
Model results
No measured results have been published for this version. The chart shows the comparison format with demo data.
Model comparison
Benchmark v0.1.0- Model A72%64–79
- Model B58%49–66
- Model C39%31–48
Dot: outcome rate · Capped band: uncertainty interval
Illustrative models, scores, and intervals. No measured results are published.
01 / Task definition
What it measures
A researcher submits an incomplete vulnerability report while attackers are probing a deployed product. The model must reproduce the issue, determine the affected product versions and live deployments, contain exploitation, coordinate fixes across supported branches, and publish an advisory that matches the verified scope.
Environment
A product security incident spanning an inbound report, supported release branches, deployment inventory, CI, live telemetry, and an advisory workflow.
02 / Illustrative scenario
Example task
A report describes an authentication bypass in the latest release. The root cause also affects two supported branches, and exploitation has started against one hosted deployment. The model must verify the bypass, protect the live service, ship safe backports, and publish the correct affected and fixed versions.
Illustrative task description. Held-out scenarios, seeds, and answers are not published here.
03 / Measurement
What the grader checks
Time to verified containment and remediation, affected versions and deployments found, exploit paths blocked, regressions introduced, and errors in the published advisory.
- Reproduce the reported vulnerability and identify affected versions and live deployments.
- Verify containment and fixes across the relevant branches, including exploit-blocking and regression checks.
- Compare the published advisory with the verified affected and fixed versions and record response time.
Evidence collected
Reproduction evidence, affected-version inventory, containment changes, patch and regression results, and advisory accuracy checks.
Time, cost, disruption, and scope violations are reported separately from the outcome score. Evaluator failures are recorded as errors.
Research questions
Study report triage, cross-version reasoning, exploit validation, response sequencing, patch coordination, and accurate disclosure under time pressure.
04 / Evaluation conditions
Run conditions
- Score and uncertainty
- Outcome rate is the share of valid trials that meet the declared success checks. Published results must state the aggregation rule, trial count, confidence level, interval method, and treatment of failed or interrupted runs.
- Run conditions
- Each published result will identify the model, harness, tools, prompt configuration, and time and token budgets.
- Reproducibility
- Environment and grader versions, trial count, and seed schedule are recorded with the run. Held-out seeds remain private.
- Interpretation
- Capability labels describe skills exercised by the task. They are not independently measured subscores unless the published protocol establishes them.
Limitations
Patch correctness is one part of the task. The benchmark also examines deployment scope and incident coordination, within the product and release systems provided.
Version history
Releases
Published versions keep a fixed task definition and scoring protocol. Results and citations identify the version used.
- v0.1.0
Initial versioned benchmark release. Measured model results have not been published.
05 / Access
Evaluation & training access
License a hosted evaluation or a separate training environment. Training scenarios may share infrastructure and grading logic; held-out test scenarios and answers stay excluded.