Industry News
Singularity-Bench Six AI R&D Questions: The inference engine written by Claude reduced the first token delay from 12 seconds to 0.48 seconds
Researchers at Carnegie Mellon University and other institutions have launched a new benchmark called Singularity-Bench. What it tests is not writing functions or fixing bugs, but whether the AI agent can build and strengthen AI R&D infrastructure by itself.
There are six questions in total, all related to AI research and development itself.
The best result is Claude Opus 5.5 with Claude Code: five wins and one draw, and the only combination with a total score exceeding the human baseline.
The most eye-catching one is the inference engine. On the same hardware, the inference engine written by Claude himself reduced the P90 first token delay from about 12 seconds of the mature engine SGLang to 0.48 seconds. In addition to faster first words, it also improves overall throughput and graphics card output through scheduling optimization.
This number needs to be viewed calmly.
SGLang is a general-purpose engine that needs to take care of various models and various loads. It is not surprising that an engine written from scratch for a specific test scenario can lead by a large margin in specified indicators. Experienced human engineers can often do this even if they perform special optimizations. What it proves is that the intelligent agent can independently complete a closed loop of engineering optimization. It does not mean that SGLang can be thrown away.
The researchers themselves also drew boundaries: the scope of the test task was determined by humans, and it was still far from true recursive self-improvement.
The direction is still worth noting. In the past, the code benchmark test was "writing software for people", but this time the test is "making tools for AI". The model is starting to work on its own upstream, which is exactly what the group of protesters outside Anthropic’s doors feared most the day it launched.
For the engineering team, a more practical inspiration is: the parameter adjustment and scheduling of the inference service may already be handed over to the programming agent for a round, and then reviewed by humans.