How to Tell When We’ve Reached AGI

When Albert Einstein and Leo Szilard grasped what their physics implied, that splitting the atom could lead to mass destruction, they didn’t rush to publish. To do so would have been to broadcast a blueprint to the world, including to those who might weaponise it first. Similarly, the work done during the Manhattan Project, at the time, went unpublished.

The implications of Artificial General Intelligence (AGI) are on the order of atomic weaponry. The ability to spin up silicon with a human-level competence means scalable military/economic/cyber supremacy.

Today, we have every big tech CEO tweeting, researchers on podcasts discussing, blogging and publishing both in America and China.

The message has continued from the beginning “it just keeps scaling”. Yet what is less often talked about is “what is on the vertical axis of that scaling plot”. The answer is: verifiable accuracy on a test set, in other words, benchmarks and exams. This is a good first step, but a far cry from the kind of intelligence that leads to supremacy.

Most practical human work does not look like an exam. In my experience it’s the opposite, stakeholders provide a set of requirements and doing it exactly to their specification can still be wrong. A system that aces the test can and will still fail in practise if all it is benchmarked to is an exam.

Ilya Sutskever, former lead scientist of ChatGPT and now founder of Safe Superintelligence, explains the loud and loose strategy from big tech like this: “…[they’re] talking about [AI] which is something that does not yet exist. You can talk about it all you want but we haven’t felt it so we can’t know”.

The signal that I’m looking for, that will tell me we’re close, is a behaviour change. CEO’s stopping their tweets, no more glossy demos, fewer papers dropped on arXiv. Expect silence, tightened access, and a lot more government involvement. The day the loudest people stop talking is the day worth paying attention to.