Harvard Mathematicians Judge AI Performance on Unpublished Research Math

Harvard’s “First Proof, Second Batch” evaluates AI performance on research-level mathematics under strict conditions. Thirty experts blind-graded solutions submitted by four leading AI systems—models from OpenAI and Google—using 10 original, unpublished problems drawn from active research (none were available in textbooks or on arXiv). Key result: the expert panel awarded passing grades on 7 of the 10 problems across the four systems tested. Earlier trial runs reportedly solved only 2 of the 10, suggesting improvements via multiple attempts or different prompting strategies, while the grading remained blind to submissions’ provenance. The organizers emphasize why unpublished problems matter: standard benchmarks often include known solution paths, but research math may involve unknown whether a solution exists at all. This second batch follows an initial evaluation conducted in February 2026, forming an ongoing framework to track whether AI performance is truly advancing at the frontier of mathematical research or merely plateauing after early benchmark gains. Overall, the exercise provides a nuanced view of AI performance: it can solve meaningful research-level tasks, but reliability is still far from uniform across problems.