Scaling law might sound familiar if you have been following the chatter around AI. Scaling law is an empirical observation that states that models become better as we train models on more data.
But the narrative has changed because we don't have infinite data or compute, aka "data wall" and "compute wall." Now, empirical results suggest that we scale out and figure out newer architectures to gain performance improvements. We must return to the drawing board and figure out novel methods to advance with AI tech. We need more science before we get more tech.
After the underwhelming performance of small models coming out of different labs (read: Databricks, Sarvam AI, Cerebras), everyone was in consensus that small models are not the way to go. Except there was still Liquid AI in the picture. See, for AI to be ubiquitous, it needs to be deployed locally on the smallest of devices, which the tech world also calls edge intelligence (an extension of edge computing). We need compute in our fridge, car steering, and a computer mouse. I mean, why not imagine the full AI armageddon, right? Liquid AI has been solving exactly that.
Raman Hasin from Liquid AI returned to the drawing board and rewrote the architecture. Going beyond the classical transformer architecture, they developed their own algorithm at MIT-- Liquid Neural Network. Transformers are brittle, rigid, and static. They don't change much once we have trained them. The weights weigh the architecture down. Once a model becomes vintage, it needs retraining. This makes them computationally expensive. Transformers are also bad at handling continuous information, like sensor data from an autonomous vehicle. So, okay, transformers have serious theoretical limitations. What do we do now?
We sit back and enjoy nature; this is precisely what the Liquid AI team would have done. They studied a worm with just 302 neurons capable of performing basic life functions like finding food and avoiding threats. It is one of the most efficient intelligence systems.
Ramin Hasani from MIT reimagined the neuron. But, how!?
Step-by-step guide on how to create a novel AI architecture from a nematode:
Transformers are static and have finite memory, i.e. at any given point they are assigned a value based on pre-calculated function. It just processes one point. To process a data stream, you must calculate values for the entire series of points. Just inefficient.
A biological neuron's state doesn't just instantly appear. It rises and falls continuously based on the rate and timing of incoming signals. Its current state is a function of its recent past. In essence, a biological neuron is a dynamic system.
And what is the mathematical language for describing dynamic systems that change continuously over time? The answer is calculus, and the specific tool is the differential equation!
So, instead of a static function, we can define the rate of change of the neuron's state (dx/dt) as a function of its current state (x) and its current input.
Simple, elegant, and efficient. It solves for discrete time limitation of transformers and provides the neuron a memory. Making the neuron like a liquid, continuously adapting to the system's state.
Now, each of our neurons is far more powerful and expressive. Thus, we need fewer of them. Lesser neurons beget lesser layers, beget lesser computing, begets edge computing, begets the next frontier of AI technology.
Disclaimer: Transformers are not truly memory-less. They have an attention mechanism, which can process a discrete data block, but they are clunky while handling this. The response time increases quadratically with the length of data they analyze thus making them unsuitable for continuous stream of data.