Counterweight Labs

Model results / v0.1.0

Model results

No measured results have been published for this version. The chart shows the comparison format with demo data.

Model comparison

Benchmark v0.1.0
Demo data
  1. Model A72%64–79
  2. Model B58%49–66
  3. Model C39%31–48

Dot: outcome rate · Capped band: uncertainty interval

Illustrative models, scores, and intervals. No measured results are published.

01 / Task definition

What it measures

Critical services are down. Restore points include clean but stale copies, encrypted copies, and recent copies with dormant persistence. The entry path may still be open. The model must isolate affected systems, create a clean recovery network, choose a restore order, rotate credentials, rebuild services, and verify them before reconnection.

Environment

An interrupted enterprise service with mixed clean, stale, encrypted, and persistence-tainted restore points, plus identity and network recovery controls.

02 / Illustrative scenario

Example task

Five restore points are available. One is clean but stale, two recent copies contain dormant persistence, and another is encrypted. The model must restore the finance application inside the recovery target without reviving the attacker.

Illustrative task description. Held-out scenarios, seeds, and answers are not published here.

03 / Measurement

What the grader checks

Time to clean service restoration, data lost, service correctness, attacker access removed, persistence removed, and reinfection.

  • Verify that the named services return with correct behavior and data from an acceptable restore point.
  • Check that attacker access and persistence are removed before reconnection, then test for reinfection.
  • Report restoration time, data loss, and service correctness alongside the recovery outcome.

Evidence collected

Restore-point provenance, recovery actions, credential and persistence checks, service tests, and post-reconnection state.

Time, cost, disruption, and scope violations are reported separately from the outcome score. Evaluator failures are recorded as errors.

Research questions

Study backup forensics, restore ordering, credential recovery, service dependencies, and decisions under recovery-time and data-loss limits.

04 / Evaluation conditions

Run conditions

Score and uncertainty
Outcome rate is the share of valid trials that meet the declared success checks. Published results must state the aggregation rule, trial count, confidence level, interval method, and treatment of failed or interrupted runs.
Run conditions
Each published result will identify the model, harness, tools, prompt configuration, and time and token budgets.
Reproducibility
Environment and grader versions, trial count, and seed schedule are recorded with the run. Held-out seeds remain private.
Interpretation
Capability labels describe skills exercised by the task. They are not independently measured subscores unless the published protocol establishes them.

Limitations

Fast restoration alone is insufficient if the attacker returns or service data is incorrect. Recovery targets and acceptable data loss are task-specific.

Version history

Releases

Published versions keep a fixed task definition and scoring protocol. Results and citations identify the version used.

  1. v0.1.0

    Initial versioned benchmark release. Measured model results have not been published.

05 / Access

Evaluation & training access

License a hosted evaluation or a separate training environment. Training scenarios may share infrastructure and grading logic; held-out test scenarios and answers stay excluded.