AI Card Bao AICARDBAO.COM

Industry News

Engineers asked fruit fly brains to “review” code, and the accuracy rate reached 88%… but when faced with difficult problems, everything went wrong.

Nick Nisi, an engineer at WorkOS, used the brain map of male fruit flies made public by Google—which contained 176,422 neurons—and ran the simulation on his laptop. He then asked this simulated “brain” to review the code submitted for review.

Fruit flies cannot read, so an intermediate layer is used in this process. An auxiliary model is designed to answer 30 descriptive questions for each code change; the answers, along with the identifiers associated with those changes, are converted into a kind of “odor” that is then presented to the olfactory system of the fruit flies. The training process relies on actual tests: successful regressions are represented as a bitter taste, while harmless changes are indicated by a sweet smell. If the fruit fly extends its proboscis after smelling this “odor,” it means it has approved the change.

Out of those 301 unfamiliar changes, it got 88.3% of them correct; whereas flipping a coin would have yielded a 50% accuracy rate.

Don’t rush to marvel. When the same set of odor data was fed into an ordinary statistical classifier, it achieved a score of 94.8%—so the fruit fly didn’t actually “win” in this case. When tested with a more challenging set of tasks involving 24 scenarios where the code behavior should have changed, the fruit fly failed to make even one correct adjustment. What it learned was merely whether there had been any change in behavior; it wasn’t able to determine whether that change was actually appropriate or not.

This is precisely the most challenging part of code review, and it is also where many automated tools encounter difficulties.

So, this can basically be considered a fun engineering joke. By the way, I’d like to remind everyone that figures like “88% accuracy” need to be understood in context—that is, compared with what, and tested on what kind of problems. The project website offers a trial version, and public pull requests can be submitted directly there. It is said that if a token is used, this fruit fly even has the ability to include its review comments within the pull request.

Related