Our research on Inductive Transformers offers the opportunity to design artificial intelligence in addition to training it. The potential outcomes include:
1. Training data efficiency
2. Continuous learning from each new data point
3. Enhanced long-range planning and reasoning
Deer babies learn to walk within ten minutes of being born. By comparison, human babies take a year to learn to walk – humans learn to walk 50,000x slower than deer! Said another way, when they’re born, deer baby brains know 50,000x more about how to use their legs than human babies do.
Human parents could perhaps be jealous of deer for learning so fast. If so, AI parents should be just as jealous of humans. Humans learn to talk and think after experiencing a few hundred million words — hearing, saying, reading, writing, thinking, dreaming. By contrast, an AI “baby” requires on the order of ten trillion words. Human babies know roughly 50,000x more about talking compared to AIs.
What if AI could learn as efficiently as humans? For one thing, data centers for training AIs today would get a lot smaller. If AI could be trained with 50,000x less data, that would take a one million square foot data center for AI training (several coming online in 2026-2027) and reduce it to the size of a closet. It would also take the ~1 Gigawatt power plant that powers such a data center (electricity for a third of Manhattan), and reduce it to the electricity used by about 13 hair dryers.
How do humans achieve this much learning efficiency? The short answer is that while an AI is nurtured on language data after being initialized (“born”), humans have language learning designed into our nature by evolution before we are born. There’s a lot to unpack about how that works...
