The GPT Pro Knockoff: What 256 Judgments Found

Correction (August 2026) — read this before any number below. This post has been re-analysed at the right unit of measurement, and most of its conclusions do not survive that. A follow-up experiment on the current GPT-5.6 Sol models is reported at the end. It reports win rates over judgments, but four judgments (two judges × two presentation orders) all read the same pair of generated outputs for one query. They are not independent samples. The effective sample size is the number of queries — eight — not thirty-two. Reporting thirty-two as the denominator overstates the precision the design carried.

Re-tested at the query level: the ChatGPT Pro comparison does not survive (7–1 with no ties, and 6–1 with one tie; adjusted p = 0.21 and 0.25); "the 4-0 cells are not coin flips" does not survive; the null-baseline update does not survive and was also contaminated. One finding is still standing — the synthesizer choice — but it fails outright under one of two defensible multiplicity corrections, it was never pre-registered as an inferential claim, it sits exactly at this design's resolution floor, and whether it survives at all depends on a multiple-comparisons decision made after seeing the data. All of this is set out in the August 2026 update at the end.

A new experiment was also run, on GPT-5.6 Sol — the model that replaced the GPT-5.5 this eval was built on — with the same architecture running against itself. The judges still produced a 5–1 query-level split (with 3 ties across nine queries), and unanimous per-query cells at a rate indistinguishable from the cells this post called "not coin flips." The instrument's noise floor is large enough to manufacture the margins this post read as results.

Correction (July 2026): Four things in this post have been repaired. (1) The update at the end — the null-baseline experiment — has had its conclusions withdrawn: the run's "single call" arm was not a control on six of its eight queries, because it executed the ensembles it was supposed to be measured against. The section is still below, with the full explanation in place. (2) The paragraph explaining where the judges' position flips concentrate asserted a cause the post's own tables contradict — flips are lowest on the widest-margin contests and highest on the closest one — so the explanation is withdrawn and the observed pattern now stands on its own. (3) The stated total cost, "roughly two hundred xhigh equivalents," did not follow from this post's own five-times and thirteen-times multipliers, which give about a hundred and forty. (4) The coordination section claimed some of its high-confidence judgments were unstable under position swap; that does not follow from the flip counts and is false against the judgment record — all four fall on queries neither judge flipped. The pre-correction wording of all four is kept in the site's private revision archive; superseded text is not republished. Three further findings from the same review — a workspace-access asymmetry between the ensembles and ChatGPT Pro, the reading of the four unanimous per-query cells, and the routing rule for contested-evidence queries — were open rather than repaired at that time (all three are resolved in the August 2026 update: the routing rule is removed, and the other two do not survive), because each changes what the post concludes rather than what it reports.

ChatGPT Pro was documented, at the time of this eval in May 2026, as access to GPT-5.5 Pro with deeper reasoning controls; OpenAI's published surface described a single model with additional thinking-time options that the standard product did not expose, and disclosed no internal multi-agent debate. The choice that is visible to the user is depth rather than breadth: more time per call rather than more concurrent calls. The result is a recognizably different shape of output, heavier on synthesis than any of the streamed reasoning models, slow enough that you go make tea while you wait for it, and good enough on hard research questions that the workspace had been quietly relying on it for several months when the question of replacing it came up.

The replacement question was practical rather than theoretical. ChatGPT Pro lives in the browser, accessed by a Chrome CDP harness that occasionally drifts from the actual web interface and sometimes drops outputs when the session times out. It is not scriptable in the way the rest of the workspace automation is scriptable, which means cron jobs that want a deep second opinion either pay the friction tax of CDP scraping or settle for a shallower model. The right shape of an answer would be a CLI tool that produced output of comparable quality without requiring a browser, ideally driven by GPT-5.5 xhigh via the same Codex CLI surface that already powers the workspace's other delegated reasoning tools.

A single xhigh call is not the same shape as ChatGPT Pro. The Mixture-of-Agents pattern proposed in Wang et al. 2024 (Duke / Together AI / Stanford / Chicago) offered an alternative reframing: instead of one model thinking longer, multiple agents thinking in parallel with their outputs aggregated through one or more synthesizer layers. The paper's actual architecture is layered (proposers feed an aggregator that can itself feed a higher-layer aggregator), but a simplified one-layer version (parallel agents + one synthesizer) is the version this build was inspired by. The paper's headline result is that a multi-agent open-source ensemble reached 65.1% on AlpacaEval 2.0, against GPT-4 Omni's 57.5%, despite each individual proposer model being weaker than the model it was beating. The pattern has empirical support for instruction following benchmarks. Whether it would transfer to the workspace's actual high-stakes decisions, where the ground truth a real reader would care about is whether an answer was useful for a specific contextual choice, was the question the build came around to answer.

Two flavors of the knockoff got built, and the eval is the story of comparing them against each other and against ChatGPT Pro itself.

Two Architectures

The first flavor is the simpler one: four GPT-5.5 xhigh agents that work blind, then a separate synthesizer that integrates their outputs after they have all finished. Each agent gets a different persona prompt (skeptic, architect, risk-analyst, empiricist) designed to cover non-overlapping cognitive ground rather than rely on temperature noise applied to the same prompt. The synthesizer reads all four outputs and produces a five-section integrated answer (consensus, disagreements with adjudication, open questions, final recommendation, and an attribution map linking each major claim back to the persona that originated it). The pattern is roughly five times the cost of a single xhigh call, and the wall clock is dominated by the slowest agent plus the synthesis step (typically nine minutes for moderate queries). The tool is called Codex Council.

The persona design is the load bearing piece of the architecture. The skeptic looks for the strongest case against the obvious answer, the architect designs the cleanest path through the problem, the risk-analyst inventories failure modes and second-order effects, and the empiricist grounds the question in established evidence and identifies cheap decisive tests. The pairs are deliberately at structural tension: skeptic and architect almost always disagree on whether to do something, while risk-analyst and empiricist sometimes disagree about whether a flagged risk is empirically calibrated. The synthesizer's job is adjudication rather than averaging, which is why the synthesis prompt explicitly references the typical structural conflicts between specific personas and asks for reconciliation rather than aggregation.

The second flavor, GPT Max, adds coordination. Same four personas, same xhigh effort, same synthesizer, but the agents exchange draft summaries between rounds via Hook Communications (HCOM), a tool that gives agents a shared SQLite event log and named addressing for sending messages to each other. The protocol is a six-phase template wrapped around each persona prompt: DRAFT, BROADCAST, LISTEN, REFINE, WRITE, EXIT. Agents work blind through DRAFT, then broadcast a three-to-five sentence summary of their position, then call hcom listen 180 to receive peer drafts, then refine their position with peer reasoning explicitly named in the final output, then write their final position, then disconnect via hcom stop. The synthesizer still runs after all four agents have finished. The pattern is roughly thirteen times the cost of a single xhigh call, and the wall clock adds about a minute of coordination overhead on top of the council's nine.

The HCOM integration was where most of the Phase 2 build time went. Codex CLI does not expose mid-turn hook injection the way Claude Code does, which meant the original plan of "messages arrive automatically between tool calls" was not implementable in the literal form intended. The fallback was structured polling: agents must call hcom listen explicitly to receive peer messages, which produces a discrete coordination model rather than the continuous-message vision. The verification round (a series of two-agent smoke tests confirming that HCOM works with codex-cli at all, that messages cross between agents within seconds, and that the sandbox configuration needs danger-full-access rather than the default workspace-write sandbox) took the better part of a day. The polling model turned out to be enough: agents read peer drafts, sharpened disagreements, and produced final outputs that cited specific peer reasoning by attribution.

Both architectures share an output directory structure so a single run from either produces files that can be compared directly. The synthesizer module is the same code, parameterized on which set of persona files to read. The CLI flag --synth gpt55|opus|both lets either architecture produce a GPT-5.5 synthesis, an Opus synthesis, or both for direct comparison. That last mode is what made the Phase 3 eval possible without doubling the agent-run cost.

The Eval

Phase 3 ran eight queries across five architectures. The four ensemble architectures are the cross of {Codex Council, GPT Max} with {GPT-5.5 synth, Opus synth}, labeled 1A, 1B, 2A, 2B. The fifth architecture is ChatGPT Pro single-call, labeled 3, which was added after the initial four-architecture run had completed and a Chrome CDP login made it possible to script Pro queries against the live web interface.

The eight queries were chosen to cover six query categories drawn from the eval methodology document: architectural decision (tool packaging defaults), strategic direction (three month focus), risk assessment of a workspace tool (GPT Max failure modes, self-referential), empirical evaluation design (the publication-review experiment), contested truth (multi-model review value), reframe-eligible (game project prioritization), and two wiki queries (AI productivity evidence in software engineering, multi-agent coordination taxonomy) included specifically to test whether the architecture preferences shifted on substantive research questions rather than workspace decisions. The wiki queries are differentiated from the other six by asking for citation-rich, broadly covering output across many primary sources rather than a decision adjudicated against workspace context.

Each architecture ran each query once with --synth both on the ensemble side and once as a single call on the ChatGPT Pro side. That produced thirty-two ensemble synthesis files (eight queries by four ensemble variants 1A, 1B, 2A, 2B) plus eight ChatGPT Pro outputs (one per query, each reused across both of the Pro contests below). By this post's own multipliers — five times xhigh for the council, thirteen for GPT Max — the total agent cost was roughly a hundred and forty xhigh equivalents, plus sixteen Opus synthesizer calls plus the eight Pro calls, with a wall clock of about four hours running the architectures sequentially.

Judging was blind A/B pairwise with position bias mitigation. The judges had no information about which architecture produced which output and were asked to evaluate on five criteria in priority order: identification of load bearing considerations, surfacing of real disagreement rather than diversity-averaging, calibration of the final recommendation, auditability of the reasoning, and surfacing of non-obvious open questions. The output schema was WINNER (A, B, or TIE), CONFIDENCE (high, medium, low), with a brief reasoning section (two to four sentences).

All eight pair contests ran the same position protocol: each pair was judged on each of the eight queries by each of the two judges in two position orderings (A-then-B and B-then-A). That produces 8 queries × 2 judges × 2 positions = 32 unique judgments per pair, and 8 pairs × 32 = 256 total judgments across the eval. The raw judgments TSV on disk has 454 rows, but 198 of those are duplicate writes of the same underlying judgment file (the eval runner appended rerun rows to the TSV without overwriting), so the working count after deduplicating by raw judgment file is 256. The duplicates are evenly distributed across the six ensemble pairs (each ensemble pair has roughly 33 GPT-5.5 rows and 32 Opus rows that collapse to 16 unique judgments per judge); the two ChatGPT Pro pairs have no duplicates because they were run after the deduplication issue had been worked around. All numbers in the rest of this post (aggregate wins, high-confidence wins, position-flip rates) are computed against the deduplicated set; an earlier summary.md of this same eval that reported pre-deduplication counts contains different numbers, and the deduplicated counts here supersede that earlier summary.

Architecture 3 was not compared against the GPT-5.5-synth ensemble variants because those were already known from the first round of pair contests to be the weaker variants of each architecture; pairing Pro against them would have wasted Pro-query budget on known-weaker baselines.

One caveat travels with every ChatGPT Pro number in the eval. The ChatGPT Pro MCP server's output extractor uses innerText rather than HTML parsing that preserves markdown, which strips <a href> attributes from Pro's hyperlinked citations. The result is that Pro's references render as plain text labels (e.g., "METR" instead of [METR/Becker 2025](https://arxiv.org/abs/2507.09089)), while the council and GPT Max architectures preserve URLs as proper markdown links via their persona outputs. This may have biased judges against Pro on wiki queries where citation density and link followability matter most to the judging criteria, but the asymmetry is present in every Pro contest, not only the wiki queries, since Pro outputs throughout the eval carry the missing-citation artifact and may also be partially de-blinded by it (a judge who notices the artifact can infer which side is Pro). Rerunning with an extractor that preserves markdown would be more apples-to-apples but is deferred.

What the Numbers Said

The eval was designed to answer three orthogonal questions: does the synthesizer choice matter, does coordination beat blind ensemble, and does the knockoff approach the real ChatGPT Pro. Each question maps to a specific subset of the pair contests, and the answers are not all the same shape.

The synthesizer is the dominant variable

The cleanest test in the matrix holds the architecture constant and varies the synthesizer: take the same four agent outputs and ask whether GPT-5.5 or Opus produces a better integration. The two pair contests that isolate this question are 1A vs 1B (Codex Council with each synth) and 2A vs 2B (GPT Max with each synth).

| Pair | Total | Wins | Opus % | |------|-------|------|--------| | 1A (cc/gpt55) vs 1B (cc/opus) | 32 | 1B 28, 1A 4 | 87.5% | | 2A (gm/gpt55) vs 2B (gm/opus) | 32 | 2B 30, 2A 2 | 93.75% |

Opus synthesis won 58 of 64 judgments on same architecture comparisons, a 90.6% win rate that holds across both architectures and is the largest effect in the entire eval. The high confidence wins skew the same direction: five high confidence judgments for 1B against zero for 1A, four high confidence for 2B against zero for 2A. Judges across both Opus and GPT-5.5 preferred the Opus synthesized outputs at rates well above a 50/50 null. The replication across two architectures is doing work here: a single 87.5% finding from one same architecture contest would have been a single data point, while the consistent 87.5% / 93.75% split across two structurally different pipelines is what licenses calling the synthesizer "the dominant variable" rather than "the dominant variable in this one architecture."

The mechanism is interpretive rather than mechanical. The synthesizer's job is adjudication of disagreements between strong arguments, and Opus appears to do that work more cleanly than GPT-5.5 on queries that turn on judgment. The output structure (consensus, disagreements with adjudication, open questions, final recommendation, attribution map) is the same; what changes is how cleanly each section gets populated. GPT-5.5 syntheses tend to flatten disagreements into hedged consensus statements where Opus would have named the disagreement and adjudicated it; they tend to produce shallower attribution maps; and they show less calibrated confidence on the final recommendation. These effects compound, and the eval cannot distinguish which of them is doing most of the work. They are also specific to one persona shape: both 1A/1B and 2A/2B used the same four-persona output set, so what the eval actually measured is whether Opus or GPT-5.5 synthesizes these persona outputs better. Whether the same synthesizer gap shows up on, e.g., temperature-noise ensembles or different persona compositions is untested.

The implication for the workspace default is direct: the cheap step that produces the largest quality gain is flipping --synth gpt55 to --synth opus. Both architectures should default to Opus synthesis, with GPT-5.5 synthesis available for eval comparisons and for applications where cost matters more than quality. The cost of the flip is one Opus call per query instead of one GPT-5.5 xhigh call, which is a fixed Opus pricing increment on top of the existing four-xhigh agent cost. For a tool that already costs five times xhigh per invocation, the increment is modest for a measurably larger quality gain.

Coordination did not beat blind ensemble

The coordination question reverses the setup, holding the synthesizer constant and varying the architecture: with the synthesizer fixed at Opus, does the coordinated GPT Max produce better outputs than the blind Codex Council? This contest is 1B vs 2B.

| Pair | Total | 1B (council) | 2B (GPT Max) | |------|-------|--------------|--------------| | 1B vs 2B | 32 | 16 | 16 |

The aggregate win count is exactly 16-16. The high confidence wins favor 1B at four against zero, which, consistent with the pattern in the synthesizer section (where the high-confidence subsample skewed the same direction as the win-count majority), would suggest the blind council outputs land more sharply than the coordinated ones on the queries where judges were confident. The reason this section does not call the result a confident preference for 1B is that the GPT-5.5 judge flips on three of the eight queries in this contest under position swap, and the Opus judge flips on one, and the eval methodology treats those query-pairs as inconclusive for the judge that flipped. The high confidence judgments themselves are not among them: all four fall on queries neither judge flipped. The honest reading is: at the aggregate level the architectures are exactly tied, with the high confidence subsample leaning toward blind council on the queries where judges were stable.

Per query, the contest is uneven in a way the aggregate hides. Each query received four judgments (two judges by two positions), and the cells below show those four judgments split between the two architectures:

| Query | Category | 1B (council) | 2B (GPT Max) | Winner | |-------|----------|--------------|--------------|--------| | Q1: tool packaging | Architectural | 0 | 4 | 2B 4-0 | | Q2: 3-month focus | Strategic | 4 | 0 | 1B 4-0 | | Q3: GPT Max failure modes | Risk (self-ref) | 2 | 2 | split | | Q4: publication-review experiment | Empirical | 1 | 3 | 2B 3-1 | | Q5: multi-model review value | Contested | 3 | 1 | 1B 3-1 | | Q6: game project prioritization | Reframe | 2 | 2 | split | | Q7: AI productivity (wiki) | Empirical/Contested | 0 | 4 | 2B 4-0 | | Q8: multi-agent coordination (wiki) | Architectural/Contested | 4 | 0 | 1B 4-0 |

The pattern that motivated the GPT Max build (coordinated agents systematically beating blind agents) does not appear. No query category lines up cleanly with one architecture: Q1 (architectural) goes to GPT Max, Q8 (also architectural) goes to Codex Council; the two wiki queries split opposite directions; the contested-truth queries split as well. The aggregate is a coin flip because the per query results are themselves split, not because the architectures are systematically equivalent on each query.

The Phase 2 plan had been built around the loose hypothesis that coordination matters, with the MoA paper as a motivating-but-not-strictly-licensing reference. MoA's actual headline comparison is layered ensemble aggregation against single-model baselines on instruction following benchmarks; the specific comparison "coordinated ensemble versus blind parallel ensemble" is not what the paper centrally tests. The build proceeded anyway on the prior that peer-aware refinement (skeptic challenging empiricist mid-task, synthesizer integrating that exchange with attribution) would be the kind of structural advantage MoA's layered aggregation gains over single-shot. The eval data does not support this prior on the eight queries tested: the 2.6× cost increase from council to GPT Max (5× xhigh to 13× xhigh) produces no detectable quality benefit at the aggregate level.

(August 2026: the reading in this paragraph does not survive — running the architecture against itself produced unanimous cells at an indistinguishable rate. See the update at the end.)

The narrower reading available in the per query data is that some queries do appear to benefit from coordination (Q1, Q4, Q7 all favor GPT Max with 4-0 or near-4-0 splits) and some are made worse by it (Q2, Q5, Q8 all favor Codex Council with 4-0 or near-4-0 splits). The 4-0 cells are not coin flips; they are unanimous-judge results that suggest a real per-query architecture effect the eval lacks the statistical power to characterize. With only eight queries spanning six categories, there are not enough samples per shape to license a "use coordination for architectural decisions, blind ensemble for strategic direction" rule. The honest reading is that the per query results contain signal that an 8-query eval cannot resolve into a confident rule, and the operational default should be the cheaper blind ensemble unless the specific query shape has a known reason to benefit from refinement.

The knockoff beat the original on workspace queries

The two ChatGPT Pro contests are 1B vs 3 (Codex Council with Opus synth versus ChatGPT Pro) and 2B vs 3 (GPT Max with Opus synth versus ChatGPT Pro). Each contest had thirty-two judgments (eight queries by two judges by two positions).

| Pair | Total | Council/GPT Max | ChatGPT Pro | |------|-------|------------------|--------------| | 1B vs 3 (cc/opus vs ChatGPT Pro) | 32 | 28 | 4 | | 2B vs 3 (gm/opus vs ChatGPT Pro) | 32 | 26 | 6 |

Both ensemble architectures beat ChatGPT Pro at rates above 80%. Across all 64 judgments against the two ensemble architectures, ChatGPT Pro had zero high confidence wins, while the ensembles collectively had twenty-five. One framing matters here: the "knockoff" is not a vendor swap. Codex Council and GPT Max use OpenAI GPT-5.5 xhigh for the four agent runs and Anthropic Opus for the synthesis layer; ChatGPT Pro uses OpenAI's deeper-reasoning Pro variant. The comparison is therefore not "alternative vendor beats incumbent" but "ensemble of OpenAI agents with cross-vendor Anthropic synthesis beats a single deeper OpenAI call." The synthesizer dominance finding above suggests the Anthropic synthesizer is doing much of the visible work.

Per query, the result sharpens further. Against Codex Council with Opus synth, ChatGPT Pro won exactly one query (Q7, the AI productivity research question, 4-0); the council won the other seven. Against GPT Max with Opus synth, ChatGPT Pro won one query outright (Q8, the multi-agent coordination wiki query, 3-1) and tied on Q3 (the self-referential failure-modes query, 2-2); GPT Max won the other six. The five workspace decision queries (Q1, Q2, Q4, Q5, Q6, excluding the self-referential Q3) were a complete sweep for the ensemble architectures in both contests.

The wiki queries are where ChatGPT Pro shows up, and the pattern across the two Opus-synth ensemble contests (the only ensemble variants Pro was compared against) is specific. Pro beats whichever of those two ensemble variants lost the 1B-vs-2B contest on that wiki query: on Q7 (where GPT Max beat council in the 1B-vs-2B matchup), Pro beats council 4-0; on Q8 (where council beat GPT Max), Pro beats GPT Max 3-1. Pro never beats the better of the two Opus-synth ensemble variants on either wiki query. The cleanest reading is that ChatGPT Pro is roughly equivalent to whichever Opus-synth ensemble variant happens to be weaker on a substantive research query, and loses to whichever is stronger. The eval does not compare Pro against the GPT-5.5-synth ensemble variants, so claims about Pro vs ensembles in general should be scoped to the Opus-synth comparison.

This is, in fairness, a narrow result. Two wiki queries is not enough to license a confident generalization about Pro's relative strength on research-shaped content. The citation-stripping caveat above also applies: the wiki queries are where citation density matters most to the judging criteria, and Pro's hyperlinked citations rendered as plain text labels almost certainly hurt its scores on this exact query type. The findings document treats Pro's performance on wiki queries as "competitive only with the underdog council variant" rather than as a stronger generalization, and that framing is what the data licenses. The knockoff premise (that four concurrent xhigh agents plus Opus synthesis would match or beat ChatGPT Pro on workspace decisions) is supported by these numbers more strongly than I expected when I designed the eval, even after acknowledging that the citation-stripping artifact may have inflated the ensemble's workspace-query margins by an unknown amount. The build was worth doing.

Judge position flip rates

Position flip in this eval is best measured per (pair, judge, query): if the same judge produces different winners on the A-then-B and B-then-A presentations of the same query, that query-pair-judge cell is flagged for that judge. Aggregating across queries for each (pair, judge) gives the count of flipped queries per pair per judge:

| Pair | GPT-5.5 flips | Opus flips | |------|----------------|-------------| | 1A vs 1B | 0/8 | 0/8 | | 1A vs 2A | 6/8 | 3/8 | | 1A vs 2B | 1/8 | 0/8 | | 1B vs 2A | 3/8 | 0/8 | | 1B vs 2B | 3/8 | 1/8 | | 1B vs 3 | 0/8 | 0/8 | | 2A vs 2B | 2/8 | 0/8 | | 2B vs 3 | 0/8 | 2/8 | | Total | 15/64 (23.4%) | 6/64 (9.4%) |

GPT-5.5 flips on roughly a quarter of the query-pair cells; Opus flips on about a tenth. The implication is consistent with the workspace's 2026-04-29 internal investigation into LLM judge position bias: GPT-5.5 has higher position bias as a judge than Opus, though not at the extreme rate an earlier (incorrectly aggregated) summary of this eval had suggested. GPT-5.5 flips most on 1A vs 2A (six of eight), then on 1B vs 2A and 1B vs 2B (three of eight each); Opus flips most on 1A vs 2A (three of eight) and 2B vs 3 (two of eight). Flips do not track the size of the measured gap, and they run against it: the widest-margin contests in the tables above flip least, with 1A vs 1B (28-4) and 1B vs 3 (28-4) at zero of eight for both judges and 2A vs 2B (30-2) at two of eight and zero, while the exact 16-16 tie between 1B and 2B flips on four of its sixteen cells. This eval does not identify what drives the position dependence; it only shows that a large quality gap is not what produces it here.

(August 2026: superseded. On a contest where both sides were the same architecture, the Opus judge picked whichever output was shown second 16 times out of 18 and reversed itself on 7 of 9 queries — so weighting it more heavily is not supported.)

The practical recommendation from this is that future evals should weight Opus judgments more heavily than GPT-5.5 judgments on close pairs, and add a third-provider judge (Gemini 3.1 Pro is the most obvious candidate) to provide a tiebreaker rather than relying on a two-model panel. The current eval treats flipped query-pair-judge cells as inconclusive for that judge, which preserves the Opus signal on the 1B vs 2B contest (where Opus flipped on only one query) while discounting the GPT-5.5 signal on the three queries where GPT-5.5 flipped. The 16-16 tie at the aggregate level survives this treatment.

What This Means for the Workspace Default

The eval was designed to produce a decision rule rather than to prove any architecture optimal, and the rule it produces is not the one the build was originally trying to validate.

(August 2026: the three paragraphs below state the post's original conclusions. Two of the three no longer hold — see the update at the end. They are left in place rather than rewritten, because the correction is the record.)

The first finding (Opus synthesizer dominates GPT-5.5 synthesizer at 90.6% on same architecture contests) is the single largest leverage point in the eval. Both Codex Council and GPT Max should default to --synth opus, with --synth gpt55 available for eval comparisons and for applications where cost matters more than quality; the cost case is the one the synthesizer section already made, a modest fixed increment on a tool that already costs five times xhigh per invocation.

The second finding (coordination does not beat blind ensemble at the aggregate level on this query distribution) means the GPT Max architecture should not be the workspace default for high-stakes decision queries. Codex Council with Opus synth produces statistically indistinguishable outputs at 2.6× lower cost, and the high-confidence subsample lean (4-0 favoring blind council) is enough of a signal to prefer the cheaper option when the choice is otherwise a coin flip. The GPT Max architecture remains worth keeping for specific use cases where coordination plausibly matters, such as queries with active methodological disagreement where the peer aware refinement step lets the empiricist challenge the skeptic's confidence and the synthesizer integrates the result with attribution. The 4-0 wins for GPT Max on Q1 and Q7 are evidence that some query shapes do benefit from coordination, even though the eight-query eval lacks the resolution to identify those shapes systematically. The operational rule is to start with Codex Council and escalate to GPT Max only when the council output feels like the agents talked past each other.

The third finding — which does not survive re-analysis; see the August 2026 update — was that the knockoff beat ChatGPT Pro on workspace queries. As written it supported the original premise of the build for the use cases the workspace actually has. The default for high-stakes workspace decisions should be Codex Council with Opus synth rather than ChatGPT Pro. The Pro browser harness should be reserved for queries where citation-dense web search across many primary sources is the actual goal, which is the cell where the wiki queries showed Pro competitive with the next best ensemble variant.

The routing rule that stood here has been removed (August 2026). It mapped query shape to method — decision queries to Codex Council with Opus, contested-evidence queries to GPT Max with Opus, broad survey research to ChatGPT Pro — and the post's own caveat two paragraphs earlier says the eval lacks the power to license category-to-architecture rules. It licensed one anyway. Nothing replaces it at the category level, because eight queries cannot support a category-level rule at any observed margin. The default that the data does support is the cheaper one: start with the blind ensemble, on cost grounds, and escalate only when you have a specific reason.

The Phase 2 build is not retired; it is repositioned from "the new workspace default" to "the niche tool for the cases where coordination plausibly matters." The Phase 1 Codex Council, originally framed as the baseline, becomes the default. The synthesizer choice, originally treated as a flag the user could pick at invocation, becomes the part of the configuration that was always going to dominate — subject to the August 2026 update below, which is where the reader should go before relying on any number in this post.

The uncomfortable observation is that I built the coordination machinery before I evaluated which variable was actually doing the work. If the synth choice eval had run first, on the Phase 1 architecture only, the 87.5% same-architecture finding would have been a single data point, strong but not enough by itself to license calling the synthesizer "the dominant variable." The 93.75% same-architecture replication on the Phase 2 architecture is what turned a single finding into a generalization. So the GPT Max build was not strictly wasted (it produced the replication that strengthens the synth-choice claim), but the coordination layer itself, considered on its own merits, is interesting infrastructure that did not produce the lift it was designed to produce. The leverage on output quality came from the integration step. I spent the budget on the input pipeline when the output pipeline was the variable that mattered.

What the Eval Did Not Settle

Three open questions remain after the eval, and they are the candidates for the next round of measurement rather than the conclusions of this one.

The null baseline question is the most consequential and the most embarrassing not to have answered. The eval compares ensembles against ensembles and ensembles against ChatGPT Pro, but it never compares ensembles against a single GPT-5.5 xhigh call. Does a single deeper call produce 80% of the value of a four-agent ensemble at 25% of the cost? If it does, the entire architecture is over engineered. A small follow-up (three queries by one xhigh call, judged against the Codex Council with Opus output for each) would establish whether the council adds value beyond a single deeper call. It is a thirty minute experiment that I had not yet run when this post was published. (The narrower thing it would settle is whether the council adds value on those queries with these judges; it would not by itself license retiring the architecture across all query shapes.) It has since been run at full scale, and the update at the end of this post carries the answer.

The query set size is enough to detect large effects (the 90.6% synthesizer dominance, the 28-4 council win over Pro contest) but not the smaller effects that would license confidence on architecture-by-query rules. The 16-16 tied result between council and GPT Max is consistent with a real fifty-fifty equivalence and is also consistent with a small but real architecture preference that the eval lacks the statistical power to detect. A future eval with twenty to forty queries (rather than eight) would be more informative about query type effects, at proportionally higher cost.

The judging panel is two models (Opus and GPT-5.5), and the position flip data shows that GPT-5.5 brings more bias than Opus and should probably be down-weighted on close pairs. A future iteration might either weight the judges (Opus 1.5×, GPT-5.5 1×) or add a third judge (Gemini 3.1 Pro is the most obvious candidate, since it would add a third provider perspective rather than another model from the same family). The position flip rate is also worth investigating directly: GPT-5.5 flipping on 23.4% of query-pair cells (versus Opus at 9.4%) is meaningful position dependence rather than pure noise, and the source of the bias (recency, length, ordering, format) is worth diagnosing before relying on GPT-5.5 as the dominant judge on future close pairs; the recommendation in this eval is to weight Opus more heavily and add a tiebreaker, not to remove GPT-5.5 entirely.

Closing

Two hundred fifty-six unique judgments is enough to detect that the synthesizer is the dominant variable, that the coordination machinery did not produce the lift it was designed to produce, and that the four-agent ensemble with Opus synthesis matches or exceeds ChatGPT Pro on the kind of workspace decision the build was originally trying to support. It is not enough to detect smaller architecture-by-query effects, and the position flip rate on GPT-5.5 (roughly a quarter of all query-pair cells) means that the most stable signal lives in the Opus judgments, which flip on roughly a tenth of cells.

The decision rule that comes out of the eval is narrower than the one the build was originally going to validate. The Phase 1 baseline becomes the workspace default. The Phase 2 coordination machinery is kept for niche cases rather than promoted to the default. The synthesizer flag, originally treated as a side option, becomes the most important configuration decision. None of these are the outcomes the project would have predicted from its own design documents, which is the right shape for an eval to produce: a result that updates the design rather than ratifying it.

The next eval question is whether the council itself is worth its cost, and the test is the null baseline run that this eval did not include. If a single xhigh call produces output of comparable quality to four agents plus Opus synthesis, the entire architecture is over engineered and the right move is to retire it. If the ensemble produces meaningfully better output than a single deeper call, the architecture earns its place. At publication I had not run that experiment, and the cost was small enough that not running it had become harder to justify than running it would be.

Update: the null baseline, run

Withdrawn (July 2026). The conclusions in this section are withdrawn pending a clean rerun. A review of the run directory found that the "single GPT-5.5 xhigh call" arm was not a control on six of the eight queries: it had full workspace access, and it used it to execute codex_council.py or gpt_max.py — the very ensembles it was supposed to be measured against — or to read the original eval's outputs. Two of the eight baseline runs open by saying so in their own first line. Only two queries are clean. The judgment arithmetic below reproduces exactly from the recorded judgments; what fails is the validity of the baseline arm, which makes this a panel-against-panel comparison on most of the queries rather than the ensemble-against-one-deep-call comparison it is described as. The companion investigation carrying the numbers is unpublished for the same reason. The section is kept here rather than deleted, and the pre-correction wording is kept in the site's private revision archive; superseded text is not republished.

Eleven weeks after publication, the missing experiment ran at full scale rather than the three-query sketch proposed above: all eight frozen queries, one GPT-5.5 xhigh call each under the same harness the council members used, judged blind against all four ensemble arms with the original protocol, plus a re-judge of the May eval's most contested pair as an anchor tying the new judgments to the old table. The complete numbers were published in a companion investigation, withdrawn as described above; the conclusions fit in three sentences.

The architecture survives its null baseline. The flagship configuration, either ensemble with Opus synthesis, beats the single call in 28 of 32 judgments for one panel and 25 of 32 for the other, with the high-confidence judgments nearly unanimous, so the recommendation this post ends on stands as written rather than corrected. What the baseline adds is a second appearance of the eval's central finding, in a comparison the original never ran: the single call is only narrowly edged by the ensembles that synthesize with GPT-5.5 and is beaten decisively by the same panels when only the synthesizer changes to Opus, which is the cleanest evidence yet that the ensemble's value concentrates in the synthesis seat rather than in the breadth of the panel. A reader who takes one operational sentence from this whole eval should take that one: before paying five times the cost for a panel of agents, spend the first increment on a better synthesizer, because on these queries the same panel barely clears one deep call when the synthesis seat holds GPT-5.5, and the synthesizer without any panel was never tested. That last comparison is the next missing experiment, and this update is me declining to claim its result in advance.

Update (August 2026): what statistical validation changed

Days after the null-baseline update was withdrawn, the whole post went back through a proper statistical treatment, and a new experiment ran to measure something this eval never measured: its own noise.

The correction that moves everything

This post reports win rates over judgments. Judgments are not independent. Four judgments — two judges, each in two presentation orders — all read the same pair of generated outputs for one query. The effective sample size for any claim about architecture is the number of queries, which is eight, not the number of judgments, which is thirty-two.

Re-analysed at the query level, with a correction for multiple comparisons:

| Claim as published | Query-level | Verdict | |---|---|---| | Synthesizer dominance (pooled) | 8/8 queries, exact cluster-permutation p = 0.0078 | conditional — survives under one multiplicity family, fails under the other; see below | | Coordination: 1B vs 2B, "16–16" | 3–3 with 2 ties; honest interval [0.25, 0.75] vs the naive [0.34, 0.66] | unresolved, and much wider than stated | | "The 4-0 cells are not coin flips" | fails, on three independent lines | withdrawn | | "The knockoff beat the original," 1B vs 3 | 7–1, adjusted p = 0.211 | does not survive | | "The knockoff beat the original," 2B vs 3 | 6–1 with 1 tie, adjusted p = 0.250 | does not survive | | Null baseline (four claims) | none separate; best adjusted p = 0.125 | does not survive, and was invalid |

The only finding still standing, and why it may not be standing at all

The Opus synthesizer beat the GPT-5.5 synthesizer on all eight queries, across both architectures. An exact permutation test at the level of query clusters gives p = 0.0078. Three things have to travel with that number.

It was not pre-registered as an inferential claim. The eval's design document named the synthesizer comparison as one of two questions, but registered no hypothesis, no test, no threshold, and no multiplicity plan — it planned descriptive rankings and warned against over-interpreting small samples. The test that validates this finding was chosen after the data existed. It is hypothesis-generating, not confirmatory.

It sits exactly at the design's resolution floor. With eight query clusters the smallest attainable two-sided p is 2/256 = 0.0078. The finding is pinned at the floor; it cannot be more significant than this at n=8. That is a fact about the instrument's resolution, not the effect's strength.

Its survival depends on a decision made after seeing the data. Five claims were corrected for multiple comparisons, and two further tests were then run on the same data outside that family. Counted as executed — seven tests — the adjusted p is 0.055 and it does not survive. Counted as a hierarchy where the pooled test replaces its two components — five tests — it is 0.039 and it does. That hierarchy was not specified in advance. Both are reported here rather than the one that reads better.

The experiment that reframes the rest: running the architecture against itself

The cleanest way to find out whether a margin means anything is to produce a margin that provably means nothing. So: take one architecture, run it twice on the same query, and have judges compare the two runs under the identical blind protocol and the verbatim original judge prompt. Two substitutions matter and are not cosmetic: the arm runs on GPT-5.6 Sol rather than the GPT-5.5 of the original eval, and the GPT judge is likewise Sol. This is a current-model replacement of the comparison, not a rerun of the GPT-5.5 one. Both sides are the same configuration. Any systematic preference is the instrument, not the architecture.

Unanimous four-of-four cells appeared on 2 of 8 queries. The contest this post called "not coin flips" produced 4 of 8. A randomisation test comparing the two rates gives p = 0.61 — not distinguishable. Unanimous cells occur at a comparable rate when there is no architecture difference at all, so unanimity is not evidence of one. (That is a sufficiency statement — noise is enough to produce these cells — not proof that noise caused the original ones. At eight queries per side there is almost no power to tell.)

At query level the same-arm contest split 5–1 with 3 ties. An architecture beating itself five to one.

And the judges, on a contest with nothing to detect: the Opus judge picked whichever output was shown second 16 times out of 18, and reversed itself under position swap on 7 of 9 queries.

The null baseline, done properly

The withdrawn version gave its "single call" full filesystem access, and on six of eight queries it used that access against itself: on five it executed the ensemble scripts it was meant to be compared with, and on one it read the original eval's own stored outputs. Only queries 4 and 7 were clean.

The rerun seals the single call: the entire workspace is replaced by an empty filesystem inside a mount namespace, so it is not hidden but absent. Sealing only the single call would swap one confound for its mirror, so the council it is compared against is sealed too — access matched on both sides, architecture the only difference.

The sealed call noticed. Unprompted, in its own answer: "Process note: I attempted the applicable codex-council workflow, but its documented CLI path was absent and two independent searches found no live executable." Every sealed cell logged filesystem denials; across 112 agent transcripts, zero delegations occurred.

Result: the council beats the sealed single call 7–1 at query level (27–5 in raw judgments), 95% interval [0.53, 0.98], n=8. An exact binomial gives p = 0.07 — the interval and the test disagree at the boundary, and the pre-registered framing is an estimate with an interval, not a test.

So the withdrawn claim's direction is reproduced. Its confidence is not: this is an estimate with a wide interval at n=8, not a demonstration.

What the noise floor does to all of it

| contest | decisive split | real difference between the sides? | |---|---|---| | same architecture vs itself | 5–1, 3 ties | none, by construction | | workspace access vs sealed | 5–2, 1 tie | the variable under test | | council vs single call | 7–1, 0 ties | the variable under test |

Both live contests sit at or barely above the margin the instrument produces with nothing to detect. Both were pre-registered as descriptive estimates rather than tests, so neither was ever going to be reported as significant. What the same-arm control adds is the scale against which to read them: the filesystem margin is indistinguishable from what the instrument produces with nothing to detect, and the architecture margin is only modestly above it.

That is the actual lesson of this post, and it is not the one it was written to deliver: the eval's noise floor was large enough to manufacture the margins it read as results, and nobody measured the noise floor because measuring it requires deliberately running an experiment you expect to find nothing in.

What this update does not establish

Four limits, stated because the results above are easy to over-read.

No ChatGPT Pro calls were made. Nothing here measures Pro's current performance. The original Pro comparison remains confounded — the ensembles had workspace filesystem access and Pro, running in a browser, did not — and that confound is not repaired by the filesystem result in this update. Pairwise win probabilities are not transitive, so a margin measured between two ensemble conditions cannot be subtracted from a margin measured against Pro.

The same-arm control may be anti-conservative. A contest between two runs of one architecture and a contest between two different architectures need not produce the same judge-agreement distribution: two genuinely different systems can differ in style or recognisability in ways that raise agreement above what two draws of one system produce. If that is happening, the noise floor measured here is an underestimate. This design cannot rule it out at n=8, and the follow-up that would is a matched control run on the second architecture.

Everything here measures judge-panel preference on these queries. There is no independent ground truth anywhere in this eval — no external correctness standard, no user outcome, no task success metric. "Better" throughout means "preferred by this panel, under this rubric, on these eight questions."

The re-analysis is not a pre-registration. The data existed before the analysis was designed. The only protection against cherry-picking is that the list of claims tested was fixed by what this post had already published — the post chose the comparisons, not the re-analysis. That is a real but narrow safeguard, and calling it more would repeat the error under examination.