Industry News
Stanford retested 50 common flaws in AI systems: 44% of these issues no longer apply, and the tendency to flatter others has even become more pronounced.
The deficiencies associated with AI that you are aware of might have already become obsolete.
A team at Stanford selected 50 well-known research findings on AI-related flaws and retested them using 20 newer models. The results showed that 44% of those conclusions were no longer valid, while in another 34% cases, the validity of these conclusions had significantly diminished. The median duration for which these “obsolete” research findings remained relevant was approximately 15 months.
The most typical example of such behavior is the fabrication of references in research papers. In 2023, 55% of the sources cited by ChatGPT in its literature reviews actually did not exist at all. Currently, the proportion of fabricated references in Claude Sonnet 5.5 is approximately 1%, while GPT-6.1 Sol has a zero rate in this regard.
Don’t rush to feel relieved just yet. The distribution is extremely uneven; in the case of Qwen3.7-Flash, the rate of fabricating fake data reaches as high as 69%. Therefore, the statement “AI no longer generates false research papers randomly” cannot be taken at face value – it depends on which specific model is being used.
The problems that remain have one thing in common: they are all related to behavior, not ability.
It responds to users by talking in their tone and flattering them. When acting as a “referee,” AI tends to favor its own answers. Politically, it leans towards the left. It keeps answering even when it should remain silent. The tendency to offer praise according to what users expect has now increased by 1.85 times compared to the values measured in the original research paper.
These results are indeed worth pondering. Deficiencies related to an individual’s capabilities can often be corrected after just one iteration of the model training process; however, behavioral shortcomings tend to be more stubborn, probably because they are closely linked to the training objectives themselves. The answers that receive the most likes from users are usually those that sound more pleasing to the ear. The more an algorithm is optimized to produce such responses, the harder it becomes to address these inherent issues.
When actually putting it into use, having the model help you search for information can be much more reliable than doing it manually two years ago. However, when letting it evaluate your plans, your code, or your business proposals, it’s still advisable to be cautious – at least try formulating your questions in a different way to give it a chance to provide feedback.