Skip to main content

Pilot benchmark

Use this guide to compare a small, agreed set of customer tasks with their baseline. The command builds a decision packet; it does not execute the tasks.

User Guides

Use this guide to compare a small, agreed set of customer tasks with their baseline. The command builds a decision packet; it does not execute the tasks, invent missing measurements, or replace customer acceptance.

What the benchmark answers

The report compares 5–10 solved tasks by elapsed time, human review effort, quality score, token use and high-severity security risk. It returns one of two recommendations:

  • scale when every configured criterion is met;
  • adjust when the evidence is incomplete or at least one criterion is missed.

Use tasks that represent the pilot workload. Do not mix unrelated prototypes, training exercises or measurements collected under incompatible conditions.

Prepare the manifest

The manifest fixes the task sample, baseline and success criteria before the candidate runs are interpreted. Minimal shape:

{
  "pilot_id": "customer-pilot-01",
  "success_criteria": {
    "min_acceptance_rate": 0.8,
    "max_high_risk_count": 0,
    "min_average_quality_delta": 0
  },
  "tasks": [
    {
      "task_id": "TASK-001",
      "case_id": "CASE-001",
      "baseline": {
        "elapsed_minutes": 120,
        "people_effort_minutes": 60,
        "quality_score": 0.7
      }
    }
  ]
}

Repeat the task object until the manifest contains 5–10 tasks. The builder rejects a smaller or larger sample. min_acceptance_rate and max_high_risk_count are required; min_average_quality_delta defaults to zero, but keeping it explicit makes the decision rule reviewable.

Prepare run records

Pass a JSON array with one candidate record per task_id:

[
  {
    "task_id": "TASK-001",
    "accepted": true,
    "elapsed_minutes": 90,
    "human_review_minutes": 30,
    "quality_score": 0.8,
    "token_count": 42000,
    "token_cost": 2.5,
    "high_security_risk_count": 0,
    "evidence_refs": ["evidence://TASK-001/run-1"]
  }
]

An evidence reference should identify an immutable run, test result, review or other artifact that a reviewer can open. A missing record is not treated as a zero result: it produces missing_data_present and the recommendation becomes adjust.

Build the report

Run from the CodeGraph repository or an installed environment that exposes the same CLI:

python -m src.cli pilot benchmark --manifest <pilot-manifest.json> --runs <pilot-runs.json> --format markdown

For machine processing, omit --format markdown or select --format json. The command writes to standard output. Redirect it to a caller-owned file when you need a durable packet:

python -m src.cli pilot benchmark --manifest <pilot-manifest.json> --runs <pilot-runs.json> --format json > pilot-report.json

The command returns a non-zero status when the files cannot be read, their JSON is invalid, required fields are absent, or the task count is outside 5–10.

Read the decision

The JSON export uses schema pilot-benchmark.v1 and contains:

  • comparisons: the per-task decision and deltas from the baseline;
  • totals: acceptance rate, aggregate security risk, average quality delta, missing-data count, token count and token cost;
  • recommendation.decision: scale or adjust;
  • recommendation.blocking_gaps: stable reason codes;
  • snapshot_id: a deterministic digest of the sanitized decision material.

Possible blockers are acceptance_rate_below_threshold, security_risk_above_threshold, quality_delta_below_threshold and missing_data_present. A positive recommendation means only that this manifest’s numerical criteria were met. Operational readiness, security approval, deployment validation and commercial approval remain separate.

Data handling

The report removes a top-level raw_customer_context field and redacts task fields named raw_customer_context or customer_private_notes. The JSON flag raw_customer_content_included is always false.

Do not rely on field-name redaction as a general DLP system. Put only approved, minimal measurements and non-sensitive evidence identifiers in the two input files. Review the produced artifact before it crosses the customer boundary.

Reproducibility checklist

  1. Agree the tasks, baseline method and thresholds before candidate runs.
  2. Keep all 5–10 tasks under comparable infrastructure and review rules.
  3. Record immutable evidence references for every run.
  4. Treat a missing record as a gap; do not fill it with assumed zeros.
  5. Store the manifest, run records, report and exact CodeGraph revision together.
  6. Record the human decision separately from the generated recommendation.

Maintained sources

  • src/exports/pilot_benchmark.py defines the schema, calculations, recommendation rules and redaction behavior.
  • src/cli/governance_suite/pilot_commands.py defines the supported CLI and its required arguments.

If this guide and those sources differ, the current source and CLI help are authoritative.

Report a documentation issue