Pilot benchmark

Use a pilot benchmark to decide which tasks to assign to CodeGraph on an ongoing basis. Compare completion time, review effort, and quality with your team’s existing process. The CLI combines the measurements into one report and checks them against criteria agreed before the pilot.

What the benchmark answers

The report compares 5–10 solved tasks by elapsed time, human review effort, quality score, token use and high-severity security risk. It returns one of two recommendations:

• scale when every configured criterion is met;
• adjust when the evidence is incomplete or at least one criterion is missed.

Use tasks that represent the pilot workload. Do not mix unrelated prototypes, training exercises or measurements collected under incompatible conditions.

Prepare the manifest

Agree on what you will measure first. Record elapsed time from task start to a finished result, human effort, and the quality scale. For effort comparisons, the report subtracts baseline.people_effort_minutes from the pilot run’s human_review_minutes. Specify which activities each measurement includes: execution, review, or both. This lets the team explain the difference and choose tasks for the next stage.

The manifest fixes the task sample, baseline and success criteria before the candidate runs are interpreted. The complete illustrative input below contains five tasks. These numbers and references are fictional; they demonstrate the report format, not customer results. Save it as pilot-manifest.json:

{
  "pilot_id": "illustrative-pilot-01",
  "success_criteria": {
    "min_acceptance_rate": 0.8,
    "max_high_risk_count": 0,
    "min_average_quality_delta": 0
  },
  "tasks": [
    {
      "task_id": "TASK-001",
      "case_id": "CASE-001",
      "baseline": {
        "elapsed_minutes": 120,
        "people_effort_minutes": 60,
        "quality_score": 0.7
      }
    },
    {
      "task_id": "TASK-002",
      "case_id": "CASE-002",
      "baseline": {
        "elapsed_minutes": 120,
        "people_effort_minutes": 60,
        "quality_score": 0.7
      }
    },
    {
      "task_id": "TASK-003",
      "case_id": "CASE-003",
      "baseline": {
        "elapsed_minutes": 120,
        "people_effort_minutes": 60,
        "quality_score": 0.7
      }
    },
    {
      "task_id": "TASK-004",
      "case_id": "CASE-004",
      "baseline": {
        "elapsed_minutes": 120,
        "people_effort_minutes": 60,
        "quality_score": 0.7
      }
    },
    {
      "task_id": "TASK-005",
      "case_id": "CASE-005",
      "baseline": {
        "elapsed_minutes": 120,
        "people_effort_minutes": 60,
        "quality_score": 0.7
      }
    }
  ]
}

Use unique task identifiers and supply 5–10 tasks. The builder rejects a smaller or larger sample. min_acceptance_rate and max_high_risk_count are required; min_average_quality_delta defaults to zero, but keeping it explicit makes the decision rule reviewable.

Prepare run records

Save the matching illustrative records below as pilot-runs.json. For a real pilot, provide measured values and accessible evidence for each unique task_id:

[
  {"task_id": "TASK-001", "accepted": true, "elapsed_minutes": 90, "human_review_minutes": 30, "quality_score": 0.8, "token_count": 42000, "token_cost": 2.5, "high_security_risk_count": 0, "evidence_refs": ["illustrative://TASK-001/run-1"]},
  {"task_id": "TASK-002", "accepted": true, "elapsed_minutes": 90, "human_review_minutes": 30, "quality_score": 0.8, "token_count": 42000, "token_cost": 2.5, "high_security_risk_count": 0, "evidence_refs": ["illustrative://TASK-002/run-1"]},
  {"task_id": "TASK-003", "accepted": true, "elapsed_minutes": 90, "human_review_minutes": 30, "quality_score": 0.8, "token_count": 42000, "token_cost": 2.5, "high_security_risk_count": 0, "evidence_refs": ["illustrative://TASK-003/run-1"]},
  {"task_id": "TASK-004", "accepted": true, "elapsed_minutes": 90, "human_review_minutes": 30, "quality_score": 0.8, "token_count": 42000, "token_cost": 2.5, "high_security_risk_count": 0, "evidence_refs": ["illustrative://TASK-004/run-1"]},
  {"task_id": "TASK-005", "accepted": true, "elapsed_minutes": 90, "human_review_minutes": 30, "quality_score": 0.8, "token_count": 42000, "token_cost": 2.5, "high_security_risk_count": 0, "evidence_refs": ["illustrative://TASK-005/run-1"]}
]

An evidence reference should identify an immutable run, test result, review or other artifact that a reviewer can open. For a missing record, the report sets decision: missing_data, adds missing_data_present, and recommends adjust. Numeric deltas use zero values for the missing record; the example below shows their effect on the totals.

Build the report

Run from the CodeGraph repository or an installed environment that exposes the same CLI. Prepare the Python environment and local configuration using the CLI guide. These examples use PowerShell from the CodeGraph repository root. Replace the quoted file placeholders with the paths to the two files you saved:

python -m src.cli pilot benchmark --manifest "<pilot-manifest.json>" --runs "<pilot-runs.json>" --format markdown

For machine processing, omit --format markdown or select --format json. The command writes to standard output. To save the report, redirect the output to a file:

python -m src.cli pilot benchmark --manifest "<pilot-manifest.json>" --runs "<pilot-runs.json>" --format json > pilot-report.json

The command returns a non-zero status when the files cannot be read, their JSON is invalid, required fields are absent, or the task count is outside 5–10.

The command returns 0 when it builds a report, including an adjust recommendation. Inspect recommendation.decision and blocking_gaps; exit status alone is not pilot acceptance. The illustrative inputs produce five comparisons, acceptance rate 1, total token count 210000, total token cost 12.5, and decision scale.

Missing whole run records are flagged, but absent numeric fields can become zero. Validate input completeness yourself, use one quality scale and cost currency throughout, and reject duplicate task_id values. Run references are not opened or verified by the builder.

Read the decision

The JSON export uses schema pilot-benchmark.v1 and contains:

• comparisons: the per-task decision and deltas from the baseline;
• totals: acceptance rate, aggregate security risk, average quality delta, missing-data count, token count and token cost;
• recommendation.decision: scale or adjust;
• recommendation.blocking_gaps: stable reason codes;
• snapshot_id: a deterministic digest of the sanitized decision material.

Each delta in comparisons is the pilot result minus the baseline. For the illustrative inputs:

The acceptance rate is the accepted task count divided by the total task count: 5 / 5 = 1 in this example. Costs are summed in the currency you used for token_cost.

Possible blockers are acceptance_rate_below_threshold, security_risk_above_threshold, quality_delta_below_threshold and missing_data_present. For adjust, review the listed reasons, complete the measurements or change the process, and repeat the comparison. For scale, use the achieved criteria and task results to agree on the next stage of the pilot.

Example: restore a missing run

Copy pilot-runs.json to pilot-missing-runs.json. Remove only the record with task_id: TASK-005; retain the other four records and a valid JSON array. Save the file as UTF-8 without a BOM. The illustrative manifest still contains all five tasks.

python -m src.cli pilot benchmark --manifest "<pilot-manifest.json>" --runs "<pilot-missing-runs.json>" --format json

The report gives TASK-005 the values decision: missing_data and missing_data: ["run_record"]. Its deltas are 0 − 120 = −120 minutes, 0 − 60 = −60 minutes, and 0 − 0.7 = −0.7 on the quality scale. These values contribute to the totals:

The reasons for adjust are quality_delta_below_threshold and missing_data_present. The command returns 0: it created a report containing the reasons for its decision.

Restore the original TASK-005 record in pilot-missing-runs.json and run the same command again. Check the table’s right-hand column and an empty recommendation.blocking_gaps list.

Data handling

The exported manifest omits its top-level raw_customer_context field. In manifest.tasks, values of raw_customer_context and customer_private_notes are replaced with [redacted]. The report includes computed task comparisons and their evidence_refs. The JSON field raw_customer_content_included is false.

Record approved measurements and non-sensitive evidence identifiers in the two input files. Review the contents of the produced artifact before it crosses the customer boundary.

Reproducibility checklist

1. Agree the tasks, baseline method and thresholds before candidate runs.
2. Keep all 5–10 tasks under comparable infrastructure and review rules.
3. Record immutable evidence references for every run.
4. Treat a missing record as a gap; do not fill it with assumed zeros.
5. Store the manifest, run records, report and exact CodeGraph revision together.
6. Record the human decision separately from the generated recommendation.

Related material

Use the CLI guide to prepare the environment and check command options. Use the scenario catalog to select pilot tasks, such as codebase onboarding, security audits, change reviews, or architecture analysis.