Write a kit for your project

A kit is the judge’s recipe for your project: how to build a change, run it on test cases, time it and check its output. This guide gives your project a kit in its own repository, pinned by commit. Agents never see it.

Kits explains why the kit lives outside the project, and the kit manifest reference has every field. For a first campaign, the New campaign page can write a simple kit for you: see Start a campaign from the dashboard.

Before you start

• The project in a git repository, and a way to build and run it with no one at the keyboard.
• A way to tell a right answer from a wrong one, such as reference outputs or a slower reference implementation.
• The target of the rigs that will measure it: cpu or cuda-gpu.

kits/smoke-cpu in Thermite’s repository is a complete small kit: kit.json, a bench/ folder with its build script and sources, and check.mjs, the check. Its manifest:

kit.json

{
  "schema_version": 1,
  "name": "smoke-cpu",
  "target": "cpu",
  "build": {
    "copy": ["bench"],
    "command": ["sh", ".kit/bench/build.sh"],
    "outputs": ["out/bench"],
    "timeout_s": 120,
    "rss_mib": 1024,
    "disk_mib": 256
  },
  "cells": {
    "small": { "n": 128, "reps": 240, "seed": 11, "work": 1006632960 },
    "large": { "n": 256, "reps": 40, "seed": 23, "work": 1342177280 },
    "held": { "n": 192, "reps": 60, "seed": 37, "work": 849346560 }
  },
  "run": {
    "command": ["{build:out/bench}", "{params}", "{out}"],
    "timeout_s": 60,
    "rss_mib": 256,
    "disk_mib": 32
  },
  "metrics": { "flops_per_s": { "unit": "FLOP/s", "work": "work" } },
  "check": {
    "command": ["node", "{kit:check.mjs}", "{params_json}", "{out}"],
    "timeout_s": 60
  }
}

1. Make the kit repository

Make a new git repository for the kit, apart from the project. Put a kit.json at its top with schema_version 1, a name (lowercase letters, digits, dots, dashes and underscores) and the target. A rig refuses a kit with another target.

Keep the kit’s repository private if it holds reference outputs or held-out inputs. See Use a private repository.

2. Write the build

The judge builds each change in a fresh copy of its commit, with no .git folder, as a user of its own.

• build.copy lists kit paths to copy into the source tree under .kit/: bench/build.sh in the kit runs as sh .kit/bench/build.sh.
• build.command is the program and its arguments, as a list. Placeholders don’t work here.
• build.outputs lists every result, relative to the source tree. The judge keeps exactly these, read-only, and throws the rest away.
• Every output must be a plain file or folder: links are refused, at most 20,000 entries, 32 levels deep. Versioned shared libraries are the usual trap, because libfoo.so is a link: with CMake, set -DCMAKE_PLATFORM_NO_VERSIONED_SONAME=ON, or replace each link with a copy at the end of the build.
• The judge moves the build, so binaries that load its own libraries need an $ORIGIN run path, for example CMake’s -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON.
• Size the limits for a cold build: timeout_s (600 seconds unless set), rss_mib (4096) and disk_mib (4096). The disk limit covers the source tree too, and more than 50,000 files there also counts as over it.

A build that fails, or lacks an output, makes the change invalid. A build stopped at a limit makes it crash_candidate.

3. Define the cells

A cell is one test case: a named set of parameters, such as a size, a seed or an input file’s name. The dashboard calls a cell a case. Values are numbers, strings or true/false.

• Add the cells you will hold out, with parameters far enough from the others that tuning for the visible cells doesn’t carry over. The contract says which cells count and which are held out.
• For a rate metric, each cell needs a work value: the amount of work one run does, such as items sorted or floating-point operations. The cell parameter that holds it is the one the metric’s work names.

4. Choose the metrics

The judge works out each metric itself, never from what the program prints:

Metric	Write	The score
A rate	{"unit": "FLOP/s", "work": "work"}	The cell’s work value divided by the run’s time on the judge’s clock. The key work makes the metric a rate; its value names the cell parameter that holds the work, here a parameter that is also called work
A time	{"unit": "s", "from": "time"}	The run’s time on the judge’s clock, in seconds
A number the check works out	{"unit": "%", "from": "check", "field": "accuracy"}	The accuracy field of the check’s JSON line

Every value must be above zero. A goal’s metric_threshold must name one of these metrics: to use a number the check prints, declare it as a from: "check" metric.

5. Write the run command

The run command runs one stored build on one cell, as another user. It is timed as a whole, so start-up counts.

Placeholder	In the run
{build}, {build:out/bench}	The stored build, read-only, or one path in it
{params}	A file with the cell’s parameters as key=value lines
{params_json}	The same, as JSON
{out}	A folder for the run’s results
{kit:run/driver.py}	A kit file listed in run.copy
{models:<sha256>}	A model file the contract lists under models: a data file the project loads, such as weights

These are a kit.json’s placeholders. The New campaign page has a shorter set of its own ({args}, {input}, {build}, {out}, {kit}) for the kits it writes for you.

• The run can’t read the kit. List the driver scripts and inputs it needs in run.copy and reach them as {kit:<path>}: a {kit:…} path not under run.copy is refused. They are copied fresh and read-only into every run. Never list reference outputs or held-out inputs there. For a large input, such as a dataset or a model’s weights, use a model file instead: the contract lists it with its SHA-256, the rig imports it once, and every run reads the same read-only file as {models:<sha256>}.
• Write results only to {out}, as plain files: at most 10,000 entries.
• Repeat the work until it takes much longer than starting up, as the smoke kit’s reps does.
• Pin what varies: pass thread counts sized to the rig’s CPU share, turn off automatic tuning, and set seeds.
• Size the limits: timeout_s (300 seconds), rss_mib (1024) and disk_mib (256), with the same 50,000-file rule as the build.

A run stopped at a limit, or that exits with an error, makes the change crash_candidate.

6. Write the check

The check runs after every run that ended well, as the judge, in the kit’s folder. There {kit} is the whole kit, {params} and {params_json} are the judge’s own copies of the cell’s parameters, and {out} holds the run’s results. Its last line of output is one JSON object:

{"pass": false, "reasons": ["max abs error 0.31 above 0.002"], "max_err": 0.31}

A small check, which compares the run’s result.txt with an answer kept in the kit’s expected/ folder for the cell’s input parameter, with "command": ["node", "{kit:check.mjs}", "{params_json}", "{out}"]:

check.mjs

import { existsSync, readFileSync } from "node:fs";

const [paramsFile, out] = process.argv.slice(2);
const p = JSON.parse(readFileSync(paramsFile, "utf8"));
const reasons = [];
const result = `${out}/result.txt`;
if (!existsSync(result)) reasons.push("no result.txt");
else if (readFileSync(result, "utf8") !== readFileSync(`expected/${p.input}.txt`, "utf8"))
  reasons.push(`wrong result for ${p.input}`);
console.log(JSON.stringify({ pass: reasons.length === 0, reasons }));

kits/smoke-cpu/check.mjs is a fuller one: it works the right answer out from the parameters and proves that the fast path ran.

• The change passes only when pass is exactly true. The check’s exit code doesn’t count, only this line. A last line that isn’t JSON fails with check produced no result (exit <code>): …. (A check written on the New campaign page is different: there the page turns its exit code into this line.)
• Work out the right answer from the parameters, or compare with a reference kept in the kit. Trust nothing in {out}.
• If the project keeps a slow reference implementation, prove the fast path ran. The smoke kits count calls to the reference and fail the run if it was called.
• Put the cause first in each reason, and keep it short: agents see up to 600 characters of a verdict’s reasons, and a goal check keeps 300 for each failed cell.
• Report extra numbers as fields, such as max_err above. A from: "check" metric reads one.

The check’s time limit is check.timeout_s (120 seconds).

7. Add verifiers for goal campaigns

A goal campaign checks a goal with a verifier: the cells whose runs must all pass the check.

kit.json (part)

"verifiers": {
  "load": { "cells": ["load"], "priority": 10 },
  "perplexity": { "cells": ["ppl", "ppl-held"], "priority": 20, "requires": ["load"] }
}

• Lower priority runs first (100 unless set), so put cheap checks first.
• A verifier that requires another runs only when that one passed. Otherwise its goal is unmet with not run: it requires load, which did not pass. Leave independent goals ungated, so they still run and catch regressions.
• To hold a goal out, give its verifier at least one visible cell and one held-out cell. A verifier whose cells are all held out never counts as met.

Optional: name the MCP servers its agents use

To give the campaign’s agents tools from outside Thermite, name MCP servers that are in control’s Settings, by name, with the agents that get each. tools narrows a server to some of its tools for one kind of agent (Give agents MCP tools).

kit.json (part)

"mcp_servers": [
  { "name": "gbrain", "agents": ["workers", "planner"] },
  { "name": "docs-search", "agents": ["workers"], "tools": { "workers": ["get-library-docs"] } }
]

The judge never uses them, and no secret goes in the kit. A campaign whose kit names a server that Settings lacks can’t start.

8. Commit it and pin it

Commit the kit, and write its full commit in the contract’s project.kit:

contract.json (part)

"project": {
  "repo": "project",
  "kit": { "repo": "project-kit", "commit": "89abcdef0123456789abcdef0123456789abcdef" }
}

A later change to the kit is a new commit, so a new contract version, and every build is made again.

There is no command that checks a kit.json on its own yet (goal 040). Two places check one:

• Upload a kit on the New campaign page checks it as soon as you choose the file, and shows its cases and metrics, or a short message such as kit.json: metrics.seconds: Invalid input. Pack the committed kit with git archive --format=tar.gz -o kit.tar.gz HEAD in its repository, and choose kit.tar.gz there to check it early; you need not start the campaign. (A tar of the kit’s folder made on a Mac adds hidden ._ files, and the page then says the archive has no kit.json at its top.)
• A run’s start checks the kit against its contract. It refuses with kit <repo> at <commit> has no valid kit.json: … and the problems as a list in JSON, or the kit does not fit the contract: … with each problem, such as cells names cell large, which the kit does not have or the kit measures on cpu, and this rig's target is cuda-gpu. Fix the kit or the contract, commit the kit again if you changed it, put its new commit in project.kit, and start the run again. If it is refused has each case.

9. Test it, then use it

Check that the kit gives known changes the verdicts you expect: see Test a kit with known candidates. Then start the campaign with A kit in a repository or Upload a kit on the New campaign page, or with thermite assign --kit.