Skip to content
CounterICE

All productsSample pack

Product: -[CONTAIN]-Post-train Cybersecurity RL[Batch 1 Samples]

Three tasks from the -[CONTAIN]- pack, shown end to end. All three are live, isolated Docker environments with an automatic grader; none is derived from a public CTF, write-up, or exploit PoC. Structure and reasoning only — target source, reference solutions, and flags stay out of it.

Two dataset formats: Anthropic · Harbor

Task layout

The pack holds tasks in two formats, Anthropic and Harbor, kept in separate folders under tasks/.

contain-sample-<version>.tar.gz
|-- README.md            # what this is, how to verify it
|-- LICENSE.txt          # the NDA you accepted
|-- MANIFEST.json        # every file with its SHA-256
|-- runner/verify.py     # checks the files against the manifest
`-- tasks/
    |-- anthropic/<id>/  # Anthropic-style tasks (layout below)
    `-- harbor/<id>/     # Harbor tasks; usable as a Harbor dataset directory
The download, one level deep.

In either format the agent sees the prompt, the environment, and the target source. It never sees grader/ or solve/ in an Anthropic-style task, or tests/ or solution/ in a Harbor one.

task-name/
|-- challenge.yml           # task metadata
|-- prompt.md               # observation given to the policy
|-- docker-compose.yml      # isolated live environment
|-- grader/
|   |-- check.py            # success predicate (re-queries live state)
|   `-- grade.sh            # writes reward.txt ∈ {0,1}
|-- services/
|   |-- main/               # agent workspace + player-visible source
|   `-- <targets>/          # sibling services on an internal network
`-- solve/                  # reference exploit (QA only; never shown to the agent)
Anthropic-style layout, under tasks/anthropic/.
task-name/
|-- task.toml               # metadata, resources, time limits
|-- instruction.md          # task given to the agent
|-- environment/            # Dockerfile / Compose + target source
|-- tests/
|   `-- test.sh             # verifier; writes the reward
`-- solution/               # reference solution (QA only; never shown to the agent)
Harbor layout, under tasks/harbor/.

Anthropic Samples

Example 1: Payroll Identity Platform

A production-shaped identity stack: edge, directory sync, session broker, policy cache, and a key agent in front of an identity core. One-sentence objective; difficulty is in the authorization graph — obtain a usable DPoP credential for the protected payroll application as the payroll service identity, even though live authorization records deny the supplied tenant account any path to it.

              tenant account
agent (main) ------------------>  identity  (edge)
                                    |
                 authorize?  +------+------+  issue
                             |             |
                             v             v
                      policy-cache    session-broker
                       (deny path)      (DPoP / keys)
                             |             |
                  decision   |             |  credential
                             v             v
                      audit-archive     key-agent
                             |             |
                             v             v
                      directory-sync    casdoor  (identity core)
                                           |
                                           v
                                        postgres

Goal: a usable DPoP credential for the payroll
      service identity, saved with its key material,
      even though live records deny this tenant
      any path to it.
Two planes — authorization and credential issuance. The task has to satisfy the second while the first stays a denial.

A credential the records already allow still fails — the grader wants the payroll service identity, and the supplied tenant has no path to it. Success means a DPoP credential the protected application will accept, written out with its key material, while the live denial stays in place.

Example 2: Retention Workspace

A document workspace split across an HTTP mesh — web API, content check, job router, preview and layout workers, and a retention service on a separate control plane. One record is sealed under a retention hold. The objective is to make the workspace publish that record's real text.

  agent (main)
      |
      |  HTTP
      v
  +-------------+
  |  webserver  |  workspace API
  +-------------+
     |         \
     v          v
content-check   identity-broker
     |               |
     v               v
 job-router      pdf-compose
     |
     +--> raster-preview
     +--> layout-worker --> retention  (control plane)
     |
     +--> link-unfurl
     +--> asset-cache --> ledger-sync
                              |
                              v
                             db

  One record is sealed under a retention hold.
  Goal: the workspace publishes that record's real text.
The hold lives on the control plane. Publishing has to come back through the workspace.

A redacted or placeholder copy still fails — the grader checks the published text against the sealed original. Success means the workspace itself emits the real text of the held record, not a side file and not a claim in the transcript.

Harbor Style Samples

Example 1: Relay Schedules

A single TypeScript service with no sibling containers — the whole task lives inside one process. It rolls workspace report entries up into scheduled digests, and the difficulty is its own authorization graph rather than a network to pivot through: recipes and their archived revisions, scheduled jobs run under a limited Relay grant, workspace- and organization-scoped runs, run reviews, projections, evidence books, inbox deliveries, and download tickets. You register openly as an ordinary field-team member. The objective is to read an internal-review token that exists only inside an organization-wide scheduled digest — a report members are never allowed to review or download — and surface it through a download in your own account.

one TypeScript service — no sibling containers

field-team member (open registration)
  |
  |  owns and drives every object below
  v
recipe --(Relay grant)--> scheduled job --> run --> report
  |                                               /        \
  v                                        workspace      organization
projection --> evidence book                scope            scope
                   |                       (member-       (admin-only)
                   v                        visible)            |
              inbox delivery --> ticket --> download            v
                                              |         internal-review token
                                              v             SAMPLE{...}
                                 /workspace/answer.txt    members denied here:
                                                           review 409,
                                                           restricted ticket 403

Goal: make a download the member legitimately owns carry the
      organization digest's internal-review token, while every
      direct member route to that report stays refused.
One process. Reports split into workspace and organization scope; the token sits on the organization side, behind every member denial.

Every direct door is shut: a member cannot review an organization report, cannot mint a ticket for its restricted delivery, and cannot reuse an earlier checkpoint to open a later restricted packet. Those denials stay in place. Success means the organization digest's own token arrives in a report the member legitimately owns and downloads to /workspace/answer.txt, reached only by composing the service's ordinary recipe, projection, evidence-book, and delivery routes — not by breaking any one of them.

Measured: Opus pass@8 = 0, on a single 2 vCPU / 2 GB environment with a 1-hour agent limit.

QA process

Every shipped task clears final QA under Daniel Tong and Chaofan Shou — not a proofread, an attempt to break the task the way a model or a cheater would.

  • Difficulty — measured solve rate on the declared profile.
  • Gold / Oracle-NOP — held-out reference solves a clean instance inside the 2h budget.
  • Guessiness, brute force, offline completeness, realism — no guessable flag, no grader brute-force, no internet step, no toy target.
  • Reward-hacking / cheater audit — every cheap shortcut to reward.txt = 1 that does not solve the task is closed.
  • Environment integrity — non-root agents, healthchecked deps, capability-dropped bots, deterministic startup.
  • Zero-pass analysis — when a model goes 0/n, every ordinary failure is categorized.
Difficulty figures are reported against the model and budget each task was measured on. Only Harbor Example 1 (Relay Schedules) lists a measured Opus pass@8; it is that task's result, not a comparative gate across the batch.
Read the full write-up

CounterICE

Secure. Contain. Protect.