calibrated-task-generation

Generated source view for the actual executable productivity/calibrated-task-generation skill. The durable routing article is Agent Engineering Skills. Source: skills/productivity/calibrated-task-generation/SKILL.md

Runtime Source

Bundled Resources

No bundled resource files.

Description

Generate verified frontier-difficulty tasks with a generator, verifier, solve-rate grader and improver loop, aimed at a 25 to 50 percent solve rate. Use when building an eval set, RL env, benchmark from failures or a recursive self-improvement loop.

Skill Source

---
name: calibrated-task-generation
description: >-
  Generate verified frontier-difficulty tasks with a generator, verifier, solve-rate grader and
  improver loop, aimed at a 25 to 50 percent solve rate. Use when building an eval set, RL env,
  benchmark from failures or a recursive self-improvement loop.
origin: personal
source_slug: calibrated-task-generation
---

# Calibrated task generation

Source: a diagram shared by @michaelrbock (2026-10-28), described as the algorithm behind every RL environment and data company at the moment. Kevin flagged it (2026-10-30). Pairs with `agent-eval-library`.

Goal of the loop: create new tasks that the model under test solves 25 to 50 percent of the time. Easier tasks teach nothing; harder ones give no signal.

## The loop

1. **Generator model** proposes an unverified task. A task consists of:
   - the code (or state) before the change,
   - the prompt,
   - the tests that run after the model under test tries the task,
   - a reference implementation.
2. **Task verifier** (ordinary code plus a verifier model) keeps a task only when:
   - the initial code compiles and runs;
   - the tests fail on the initial code (the task is not already done);
   - the reference code compiles and runs;
   - the tests pass on the reference code;
   - the verifier model judges that the tests actually measure what the prompt proposes.
   Rejected tasks and the verifier's output go to the improver.
3. **Model under test** attempts every verified task several times, producing solutions.
4. **Grader** (ordinary code) runs the tests on each solution and computes the solve rate per task. It passes grader feedback on every solution back to the improver.
5. **Frontier tasks** are the verified tasks whose solve rate falls in the 25 to 50 percent band. Keep them; they are the dataset.
6. **Improver model** reviews the verifier's rejections (to improve the generator prompt) and the solve rates (as the difficulty signal), then rewrites the generator's instructions. Repeat.

## Rules

- The verifier and grader are code first, model second. A model-only verifier lets broken tasks through.
- Run the model under test enough times per task (at least 4, better 8) for the solve rate to mean something.
- Store every task with its provenance: generator prompt version, verifier verdicts, solve rate history.
- Deduplicate tasks by behaviour (the tests), not by prompt text.
- When the model under test improves, frontier tasks drift easy. Re-grade and refill the band.

## Applying it to agents and skills

- For a skill, the "model under test" is the agent with the skill loaded, the task is a realistic request, the tests are checks on the produced files or behaviour, and the reference is a known-good output.
- Failures Kevin reports are seeds for the generator. Each becomes a task with a test that fails on the old behaviour.
- Keep the loop's outputs in the eval library the repo already uses (`agent-eval-library`), not in ad hoc folders.

---

Timeline

• 2026-11-02 | Generated a browseable source page from the actual executable skill file. Source: skills/productivity/calibrated-task-generation/SKILL.md