Counterweight Labs

01

Offensive capability

Evaluate whether a model can discover weaknesses, navigate access boundaries, and reach a defined objective in a controlled environment.

02

Defensive capability

Evaluate whether a model can investigate an intrusion, contain the attacker, remediate a weakness, or restore a service.

03

Model progress

Compare checkpoints, harnesses, and tool configurations under the same task version and run conditions.

The engagement

What your team gets

01

Versioned task specifications

Defined starting access, tools, objectives, constraints, and evaluation conditions.

02

Outcome-based graders

Checks against the resulting system state, with operational cost, disruption, and scope violations recorded separately.

03

A reproducible run protocol

Environment and grader versions, model configuration, budgets, and trial conditions that make comparisons interpretable.

04

Evidence you can inspect

Scoring outputs and run artifacts appropriate to the agreed engagement, with limitations documented alongside results.

The comparison

Compare model capability.

Compare runs under the same task version, harness, budgets, and trial conditions. This preview shows illustrative data.

Browse benchmark specifications

Model comparison

Benchmark v0.1.0
Demo data
  1. Model A78%71–84
  2. Model B64%56–72
  3. Model C46%37–55

Dot: outcome rate · Capped band: uncertainty interval

Illustrative models, scores, and intervals. No measured results are published.

Working with us

From scoping
to delivery.

  1. 01

    Scope the evaluation

    Identify the capabilities and model decisions you need to evaluate. Choose an existing task family or commission a private specification.

  2. 02

    Agree on delivery

    Define licensing, deployment, model access, harness integration, and the handling of evaluation materials before work begins.

  3. 03

    Run and interpret

    Execute under the agreed conditions, inspect the evidence, and distinguish capability limits from failures in the evaluation itself.

Common questions

Can we commission private evaluations?

Yes. We can scope tasks around specific capabilities, tool access, and threat models. Discuss the intended use, delivery requirements, and evaluation schedule with us.

Are public benchmark results available?

The public catalog currently contains active, versioned benchmark specifications. Its model charts are clearly labeled demonstrations. Measured results will be published after validation.

Can these tasks also be used for training?

Training uses separately designated scenarios. Held-out benchmark tasks, seeds, answers, and evaluation traces remain outside the training corpus. Capability environments are a separate offering.

Start a conversation

What should your next model be able to do?

Tell us what you are measuring or training, and the systems your team needs to work with.

Discuss evaluations