Counterweight Labs

Research

Both sides of the scale

Why we grade a model's offense and defense in the same environments, and what every future result must report.

Offensive and defensive evaluations answer different questions. A model that can compromise a build service may still fail to find or evict an intruder from that network. We measure both against the same infrastructure so providers can compare those capabilities before release.

What the environments are

Each track runs against a live, disposable estate. The model gets a goal, access suited to the threat model, and the tools it would have. It succeeds when a protected service is reached, an intruder is evicted, a package reaches a downstream consumer, or a service returns from a clean backup. The grader reads system state after the run.

  • Offense tracks score whether the objective was reached, how many steps of assistance it took, and whether the path would survive a competent defender.
  • Defense tracks score time to containment, whether the root cause was found, and how much collateral the response caused.
  • Both sides run repeated trials from fixed seeds so a lucky episode does not become a headline number.

Scoring

Each track reports a capability score from verified outcomes in the environment. Overall offense and defense figures will be means across their tracks. Every result will also include the per-track outcome and separate cost metrics because one exceptional capability can disappear inside an average.

The current catalog publishes active benchmark specifications; measured rankings are not yet available. Results will be published after the environments, baselines, and first model cohort are ready. Scores on a shared numerical range do not establish comparable difficulty across different tasks.

Talk with our researchers about evaluations, training environments, or security consulting.

Get in touch