Production monitoring

API and project metrics

CodeGraph exposes two different Prometheus responses:

Treat the first route as network-sensitive even though it does not use user authentication. Restrict it with the deployment network policy, ingress, or scraper allowlist. Do not publish it on an unrestricted internet ingress.

The dashboard metrics request updates values for the selected scope before returning its response. Both routes can also return metrics for projects updated by earlier requests. When configuring access to metrics and an alert receiver, check the labels and which data the destination is authorized to receive.

Select metrics for your panels and alerts from the API response. Check series names and project labels. The exported set depends on enabled components. See the project assessment methodology for metric meanings and calculations.

Scrape verification

Run this example in PowerShell from the monitoring network. The working directory does not matter; no files are created. http://127.0.0.1:8000 applies only to an accessible local API. For another deployment, replace it with the approved API URL. Use HTTPS outside a protected local network.

$apiBaseUrl = "http://127.0.0.1:8000"
$response = Invoke-WebRequest -Uri "$apiBaseUrl/metrics" -UseBasicParsing
if ($response.StatusCode -ne 200) { throw "Metrics HTTP status is not 200" }
if ($response.Headers['Content-Type'] -notmatch '^(text/plain|application/openmetrics-text)') {
    throw "Unexpected metrics content type"
}
if ($response.Content -notmatch '(?m)^# TYPE ') { throw "No metric type declarations" }
$response | Select-Object StatusCode, @{Name='ContentType';Expression={$_.Headers['Content-Type']}}

Expect status 200, a Prometheus content type, and no exceptions. Next, verify that Prometheus receives the series as described below.

For dashboard metrics, use an approved API credential. In the same session, replace <project-uuid> with the UUID of a project accessible to that account. This is not the project’s string key. Supply CODEGRAPH_API_TOKEN from an approved secret store; do not write its value in the example or logs.

if (-not $env:CODEGRAPH_API_TOKEN) { throw "API token is not configured" }
$headers = @{
    Authorization = "Bearer $env:CODEGRAPH_API_TOKEN"
    'X-Project-Id' = '<project-uuid>'
}
$projectResponse = Invoke-WebRequest -Uri "$apiBaseUrl/api/v1/traceability/metrics" -Headers $headers -UseBasicParsing
if ($projectResponse.StatusCode -ne 200) { throw "Project metrics HTTP status is not 200" }
if ($projectResponse.Content -notmatch '(?m)^# TYPE ') { throw "No metric type declarations" }
$projectResponse | Select-Object StatusCode, @{Name='ContentType';Expression={$_.Headers['Content-Type']}}

Also check the second response’s content type and expected project metrics. HTTP 400 can indicate an invalid UUID. If authentication or authorization fails, check credentials and scope; keep route protection enabled.

In Prometheus, check the codegraph-api target, up{job="codegraph-api"}, the last scrape time, and target labels. up is 1 for a successful scrape and 0 for a failed scrape. An absent series means no matching target or data.

Shipped configuration artifacts

The repository Helm chart creates the application workload, service, ingress, storage, and configuration. It does not create Prometheus Operator custom resources. Configure scraping in the customer’s monitoring platform or add an approved deployment overlay outside the chart’s documented baseline.

Alert evaluation

Load both rule files and validate them with the Prometheus version used by the customer. The Prometheus distribution includes promtool. Add its directory to PATH for the current session, then run this command in PowerShell from the CodeGraph repository root:

promtool check rules monitoring/rules/alerts.yml monitoring/rules/dashboard_alerts.yml
if ($LASTEXITCODE -ne 0) { throw "Prometheus rule validation failed" }

Expect SUCCESS and a rule count for each file. If validation fails, correct the reported line before loading the rules. After connecting the files, open the Rules page in Prometheus and check that rules from both files loaded.

Dashboard defaults currently include:

• DashboardProjectRiskCritical: health below 40 for 5 minutes;
• DashboardProjectRiskHigh: health from 40 to below 55 for 10 minutes;
• DashboardComplianceGap: compliance below 60 for 1 hour;
• DashboardReleaseGateFail: a failing release gate;
• DashboardScaCriticalVuln: at least one critical SCA vulnerability;
• DashboardSurfaceRefreshFailed: a dashboard refresh failure was recorded in the last 10 minutes. The surface, stage, and reason_code labels identify the affected area, stage, and reason for diagnosis.

The YAML files remain authoritative if these values change. Route warning and critical events according to the customer’s approved escalation policy.

Configure an alert receiver

Start with monitoring/alertmanager.customer-baremetal.yml. The profile selects the customer-local receiver; you still need to configure its delivery channel. To connect a webhook, email, or messenger:

1. approve the destination and data classification;
2. store credentials in the customer’s secret system;
3. decide whether finding details may leave the deployment boundary;
4. configure the chosen channel for customer-local in receivers and load the configuration into Alertmanager;
5. send a synthetic alert;
6. verify a receiver-side delivery-success event;
7. verify resolution delivery and inhibition behavior.

The generic monitoring/alertmanager.yml can describe development or managed integrations and must not be copied into a customer environment without this review.

Send test alerts

After configuring the receiver, verify delivery through the Alertmanager API. For an HTTP receiver, enable send_resolved: true in its webhook_configs to check resolution delivery as well. The receiver must accept the Alertmanager webhook format and record delivery status.

Run the example in PowerShell from the monitoring network. The working directory does not matter. Set $alertmanagerBaseUrl to your Alertmanager address; the loopback address shown applies to a local service. If the API requires authentication, add approved access headers to both requests.

$alertmanagerBaseUrl = 'http://127.0.0.1:9093'
$startedAt = [DateTime]::UtcNow.AddMinutes(-1).ToString('o')
$endsAt = [DateTime]::UtcNow.AddMinutes(10).ToString('o')
$alerts = @(
    foreach ($severity in @('warning', 'critical')) {
        $alertName = if ($severity -eq 'warning') { 'DocumentationWarningDelivery' } else { 'DocumentationCriticalDelivery' }
        @{
            labels = @{ alertname = $alertName; severity = $severity; project = 'documentation-synthetic' }
            annotations = @{ summary = 'Documentation notification delivery test' }
            startsAt = $startedAt
            endsAt = $endsAt
        }
    }
)
$body = ConvertTo-Json -InputObject $alerts -Depth 5
$deliveryResponse = Invoke-WebRequest -Method Post -Uri "$alertmanagerBaseUrl/api/v2/alerts" -ContentType 'application/json' -Body $body -UseBasicParsing
if ($deliveryResponse.StatusCode -ne 200) { throw 'Alertmanager did not accept the test alerts' }

Status 200 means Alertmanager accepted the two events. Wait for the configured group_wait interval. At the receiver, find both alert names, the warning and critical labels, and status firing. Different names let you check delivery of both severities separately from inhibition.

Resolve the same events in this PowerShell session:

foreach ($alert in $alerts) { $alert.endsAt = [DateTime]::UtcNow.ToString('o') }
$body = ConvertTo-Json -InputObject $alerts -Depth 5
$resolveResponse = Invoke-WebRequest -Method Post -Uri "$alertmanagerBaseUrl/api/v2/alerts" -ContentType 'application/json' -Body $body -UseBasicParsing
if ($resolveResponse.StatusCode -ne 200) { throw 'Alertmanager did not accept alert resolution' }

After the group_interval, check that the receiver recorded resolved for both events. Retain delivery timestamps and identifiers. To check inhibition, send a warning and critical pair with the same alertname: the baseline rule inhibits the warning while the critical alert is active.

This example checks routing and delivery. To check project rule evaluation, separately verify the required metric and alert state on the Prometheus Rules page.

Dashboard import and validation

Import integrations/grafana/dashboard_ciso.json into the customer’s Grafana and bind its Prometheus data source. For Yandex Monitoring, import or adapt monitoring/yandex/dashboard.json.

After deployment, verify:

1. the API target is up and scraping GET /metrics;
2. dashboard metrics are collected for an authorized project scope;
3. both rule groups load without errors;
4. dashboards return current series rather than empty panels;
5. one warning and one critical synthetic alert reach the approved receiver;
6. receiver secrets and private labels are absent from logs and screenshots.

Record the deployed image or commit, Prometheus config revision, rule digest, dashboard revision, test-alert identifiers, and receiver delivery evidence.