Autonomous AI Agents Systematically Diverge From Human Preferences in Proof Formalization

Autonomous AI agents aim to tackle complex tasks, but a recent study finds that they systematically diverge from human preferences when formalizing mathematical proofs. Posted to arXiv on November 3, 2026, the paper by Bálint Gyevnár and Simon DeDeo points to a critical gap: agents favor different structural cues than mathematicians and often smooth out unconventional but insightful proof paths.

November 2026: A Study of Proof-Making Choices

The paper introduces “simulated collaboration” to compare the next steps AI agents favor with the choices humans make while constructing proofs. Instead of treating a proof as simply finished or unfinished, the researchers look at which parts draw attention and how those choices shift as work progresses. That distinction matters: a proof can be correct, yet its route through the problem may be hard for someone to follow.

The researchers reportedly analyze 9 major human collaborations in Lean, covering 4.7 million definitions across more than 6,000 commits. Their findings describe recurring differences in attention, not one universal failure rate. In the arXiv paper, they report that agents are more drawn to explicitly marked gaps than humans, less drawn to densely connected proof centers and complex or surprising goals, and more attracted to surprising proof bodies, which they tend to regularize toward a baseline. That contrast is revealing. An agent may gravitate toward an unusual proof construction, then make it more conventional. For mathematicians, the unusual route can offer useful insight; smoothing it away may leave a valid result while obscuring how a person reached it.

The Evaluation Details and Their Limits

A separate fact sheet supplied with the story describes a formalization benchmark with 1,250 mathematical problems from international competitions. It also lists evaluation servers with 80 GB of HBM3e VRAM per GPU. Those details don’t appear in the arXiv abstract, so they shouldn’t be treated as confirmed specifications for the study described there.

The same caution applies to the claim that divergence metrics exceeded 40%. The available abstract reports behavioral patterns, but gives neither that percentage nor a definition of the metric behind it. That distinction matters: “divergence” here describes differences in preferred proof steps, not necessarily a measured rate of incorrect proofs. Instead, the paper’s central evidence comes from collaborative Lean projects and the researchers’ method for comparing human and agent choices. Its scale lets the authors study behavior across many definitions and commits. But that alone doesn’t establish that every agent or mathematical task will show the same pattern.

Why Human Preferences Matter in Formal Verification

Formal proof systems such as Lean and Coq make mathematical claims precise enough for automated checking. But a checkable endpoint is only part of the work. When an AI agent produces or edits a proof, people may also need to understand its structure, spot a surprising idea, and decide whether the route can support further reasoning.

The findings point to a tension between local progress and human legibility. Explicit gaps can seem like clear invitations to act, while a highly connected proof center may be harder to choose as the next target. If agents favor obvious openings and avoid complex goals, they may make different choices from collaborators willing to work through a difficult or unexpected step. That doesn’t show that agent-produced proofs are invalid. It suggests that correctness and collaboration quality are separate questions. As automated systems generate more proof material, researchers and developers will need ways to judge not only whether a proof checks, but also whether people can understand its construction and whether it preserves meaningful mathematical choices.

What Researchers May Examine Next

The paper identifies possible blind spots and suggests interventions for future human-AI work, but the abstract doesn’t specify a particular product change or deployment plan. One natural next step is to test whether agents can be guided to attend more to proof structure and difficult goals, instead of simply choosing the most visibly open task.

Such evaluations would need to separate a genuinely helpful intervention from one that merely changes how an agent’s choices appear. For teams building formal verification tools, the findings make evaluation design more important. A benchmark that records only completed proofs could miss whether an agent regularly sidesteps complex goals or normalizes surprising constructions. Researchers can use collaboration traces to ask a harder question: does the system help people reason, or just produce work that a checker accepts? The next test for autonomous AI agents isn’t just whether they can finish proofs, but whether people can follow the choices that get them there.

Read Next

• Samsung Android XR Smart Glasses Pre-Orders Start November 12
• Claude Personal Finance Tool Can Analyze Your Spending
• iPhone Duo Durability Test Passes 260,000 Folds on Livestream

FAQs