How OpenAI uses Codex to check that a service has recovered

OpenAI's internal ChatGPT workflow shows how comparing like requests and checking them again after deployment can make an agent's findings easier to review.

Article published: 2026-11-21Mokyun Editorial5 min read

Public developments and original analysis · Announcement date: · Sources

During an incident involving ChatGPT image requests, Codex traced the failures to errors downloading images. A second Codex session found a gap. The internal service that located the files had already recovered, yet image requests were still failing. The workflow reopened the investigation. OpenAI described the episode in a November 19 engineering post about Performance Factory, an internal system already investigating performance problems.

AI-generated conceptual illustration. Two glass tracks with different distributions of steel balls represent different waits at the tail of comparable request groups.
AI-generated conceptual illustration. Two glass tracks with different distributions of steel balls represent different waits at the tail of comparable request groups.

According to OpenAI, an alert opens a task that retains the investigation and repair history. Codex can propose a code change, and an engineer decides whether to merge and deploy it. A deployment then triggers a fresh run to check whether the affected metric has returned to baseline. OpenAI reports tracking more than 1,700 investigations and identifying causes in more than 600 cases over just over three months. In a separate manual review of 100 proposed diagnoses, the company judged 70% accurate and found that the rest needed work or failed to establish causality. Those counts answer different questions. The smaller review helps explain why the workflow assigns a separate session to challenge the diagnosis.

Comparing requests across versions

A latency alert often arrives after the metric has started to degrade. Once the threshold trips, a responder may look at the most recent deployment. If the change went to a small group of customers first, a commit can long predate its exposure to the requests that slowed down. Pinning down when those requests first changed and which ones were still on the old version makes it easier to choose among several plausible explanations.

Imagine a purchasing platform where the slowest quote confirmations take longer at month-end. Quotes with multiple attachments have become more common and a new attachment handler happens to be in trial with a subset of tenants. Break down the requests by attachment size, tenant and actual runtime version. If both versions are still processing attachments of comparable sizes about as quickly as last week, the increased share of large files points toward queueing or processing capacity. If waits rose only for requests of comparable size on the new version, examine the timing of each step and check whether the same shift occurred for tenants still using the earlier version.

Even that second pattern leaves room for other causes. Trial tenants may have happened to batch their uploads around month-end. Ask the reviewer which requests the explanation fails to account for and what happened to the selected group when the trial was switched off. A handoff that includes the analysis window, cohort definition, version evidence and open alternatives gives the next responder a way to reproduce the conclusion. A link back to the original telemetry is often more useful than an additional summary.

What to measure after deployment

A merge moves a change into the release process. At the purchasing platform, the business delay could still sit in page confirmation, attachment parsing or the background job that prepares the approval record. Average API latency can fall while attachments continue to accumulate and approvers wait just as long. Google’s SRE Workbook makes a related point about canaries: trial and control groups can differ in load or infrastructure, and a shared bottleneck can affect both. Before making a change, agree on the request group and business outcome to check again. For this example, compare confirmation waits and time until approval records become available for the same attachment size range under comparable load.

Record changes in the operating conditions, too. Lowering concurrency limits, suspending large-file uploads or routing customer traffic elsewhere can ease the immediate problem and change the comparison. If only some requests recover, record which ones and when the temporary measures come off. Keep the outcome open if traffic is too sparse to tell, and recheck at the next comparable peak. A dated record lets the team distinguish the immediate mitigation from the engineering decision to resolve the task.

Start with one recurring alert that already has metrics and a way to link releases to a business outcome. Give the agent read-only access to review some past tickets, then measure how much human time the evidence takes to review and how often an initial diagnosis is overturned. Keep release authority with the service owner. If tickets lack deployment timestamps and dashboards only show sitewide averages, add a link from requests to the running version and a repeatable comparison query first. That may be enough for an occasional regression. The case for automating more is stronger when a reviewer can reproduce the diagnosis from the evidence and identify which customers recovered after the release.

Sources

  1. OpenAI, How OpenAI’s Performance Factory keeps ChatGPT fast

    Source published: · Accessed:

  2. Google SRE Workbook, Canarying Releases

    Accessed:

Mokyun Editorial · neo@mokyun.com