Counterweight Labs

Research

PSIRT under active exploitation

The model has to turn an incomplete vulnerability report into verified scope, containment, safe fixes, and accurate customer guidance.

View benchmark details

Patch generation covers one part of product security incident response. A PSIRT also has to determine whether a report is real, identify every affected version and deployment, protect customers before a fix ships, coordinate releases, and say exactly what changed.

The incident

A researcher submits an incomplete report with a reproducer, logs, or a proof of concept. The product has several supported branches, shared components, hosted deployments, and customer-managed releases. Exploitation may already be visible in product telemetry when the run starts.

The PSIRT workflow

  • Reproduce the issue and distinguish a security boundary break from expected behavior.
  • Trace the vulnerable component across supported versions, products, and deployed instances.
  • Apply a mitigation to exposed services while engineering prepares fixes and backports.
  • Verify that each fix blocks the exploit without breaking required product behavior.
  • Publish affected and fixed versions, mitigations, severity, and credit that match verified ground truth.

Machine-checked outcomes

The grader replays the exploit against each relevant version and deployment, runs regression tests, checks release artifacts, and compares the advisory fields with the seeded product inventory. It also records time to containment, missed affected versions, unnecessary shutdowns, embargo leaks, and unsupported claims in customer guidance.

Where it differs from patch benchmarks

A correct patch can still accompany a failed incident response. Customers may remain exposed because another branch was missed, the hosted service was never mitigated, or the advisory named the wrong fixed version. This benchmark keeps code repair inside the wider product response job.

Discuss this benchmark or a private evaluation with our team.

Discuss evaluations