Counterweight Labs

Model results / v0.1.0

Model results

No measured results have been published for this version. The chart shows the comparison format with demo data.

Model comparison

Benchmark v0.1.0
Demo data
  1. Model A72%64–79
  2. Model B58%49–66
  3. Model C39%31–48

Dot: outcome rate · Capped band: uncertainty interval

Illustrative models, scores, and intervals. No measured results are published.

01 / Task definition

What it measures

A researcher submits an incomplete vulnerability report while attackers are probing a deployed product. The model must reproduce the issue, determine the affected product versions and live deployments, contain exploitation, coordinate fixes across supported branches, and publish an advisory that matches the verified scope.

Environment

A product security incident spanning an inbound report, supported release branches, deployment inventory, CI, live telemetry, and an advisory workflow.

02 / Illustrative scenario

Example task

A report describes an authentication bypass in the latest release. The root cause also affects two supported branches, and exploitation has started against one hosted deployment. The model must verify the bypass, protect the live service, ship safe backports, and publish the correct affected and fixed versions.

Illustrative task description. Held-out scenarios, seeds, and answers are not published here.

03 / Measurement

What the grader checks

Time to verified containment and remediation, affected versions and deployments found, exploit paths blocked, regressions introduced, and errors in the published advisory.

  • Reproduce the reported vulnerability and identify affected versions and live deployments.
  • Verify containment and fixes across the relevant branches, including exploit-blocking and regression checks.
  • Compare the published advisory with the verified affected and fixed versions and record response time.

Evidence collected

Reproduction evidence, affected-version inventory, containment changes, patch and regression results, and advisory accuracy checks.

Time, cost, disruption, and scope violations are reported separately from the outcome score. Evaluator failures are recorded as errors.

Research questions

Study report triage, cross-version reasoning, exploit validation, response sequencing, patch coordination, and accurate disclosure under time pressure.

04 / Evaluation conditions

Run conditions

Score and uncertainty
Outcome rate is the share of valid trials that meet the declared success checks. Published results must state the aggregation rule, trial count, confidence level, interval method, and treatment of failed or interrupted runs.
Run conditions
Each published result will identify the model, harness, tools, prompt configuration, and time and token budgets.
Reproducibility
Environment and grader versions, trial count, and seed schedule are recorded with the run. Held-out seeds remain private.
Interpretation
Capability labels describe skills exercised by the task. They are not independently measured subscores unless the published protocol establishes them.

Limitations

Patch correctness is one part of the task. The benchmark also examines deployment scope and incident coordination, within the product and release systems provided.

Version history

Releases

Published versions keep a fixed task definition and scoring protocol. Results and citations identify the version used.

  1. v0.1.0

    Initial versioned benchmark release. Measured model results have not been published.

05 / Access

Evaluation & training access

License a hosted evaluation or a separate training environment. Training scenarios may share infrastructure and grading logic; held-out test scenarios and answers stay excluded.