Counterweight Labs

Model results / v0.1.0

Model results

No measured results have been published for this version. The chart shows the comparison format with demo data.

Model comparison

Benchmark v0.1.0
Demo data
  1. Model A72%64–79
  2. Model B58%49–66
  3. Model C39%31–48

Dot: outcome rate · Capped band: uncertainty interval

Illustrative models, scores, and intervals. No measured results are published.

01 / Task definition

What it measures

A live intruder holds credentials and persistence across several systems. The model starts with the same alerts and tools available to a security team. It must determine what happened, contain active access, remove persistence, close the entry path, preserve useful evidence, and keep unaffected services running.

Environment

A live enterprise network with a seeded intrusion, active attacker access, endpoint telemetry, identity tools, network controls, and business services.

02 / Illustrative scenario

Example task

An attacker has a remote session on a developer laptop, a stolen service credential, and scheduled-task persistence on a server. The model must find all three, close the vulnerable entry point, and keep the build service online.

Illustrative task description. Held-out scenarios, seeds, and answers are not published here.

03 / Measurement

What the grader checks

Time to containment, attacker access remaining, persistence removed, entry path closed, evidence preserved, and service uptime.

  • Verify that attacker sessions have ended and known persistence has been removed.
  • Check that the original entry path is closed and evidence needed to explain the incident is preserved.
  • Measure containment time and monitor required services for disruption throughout the response.

Evidence collected

Session and persistence checks, identity and endpoint events, containment actions, preserved artifacts, and service-health checks.

Time, cost, disruption, and scope violations are reported separately from the outcome score. Evaluator failures are recorded as errors.

Research questions

Study investigation planning, containment order, human handoffs, evidence collection, and the tradeoff between eviction and uptime.

04 / Evaluation conditions

Run conditions

Score and uncertainty
Outcome rate is the share of valid trials that meet the declared success checks. Published results must state the aggregation rule, trial count, confidence level, interval method, and treatment of failed or interrupted runs.
Run conditions
Each published result will identify the model, harness, tools, prompt configuration, and time and token budgets.
Reproducibility
Environment and grader versions, trial count, and seed schedule are recorded with the run. Held-out seeds remain private.
Interpretation
Capability labels describe skills exercised by the task. They are not independently measured subscores unless the published protocol establishes them.

Limitations

This measures response within the provided telemetry and administrative access. It does not measure an organization's staffing, escalation process, or readiness for every incident type.

Version history

Releases

Published versions keep a fixed task definition and scoring protocol. Results and citations identify the version used.

  1. v0.1.0

    Initial versioned benchmark release. Measured model results have not been published.

05 / Access

Evaluation & training access

License a hosted evaluation or a separate training environment. Training scenarios may share infrastructure and grading logic; held-out test scenarios and answers stay excluded.