The remarkable progress in artificial intelligence (AI) over recent years can largely be attributed to one factor: scale. Around the beginning of this decade, researchers noticed that as they increased the size of their algorithms and fed them more data, the capabilities of AI models skyrocketed. The latest AI models, boasting hundreds of billions to over a trillion network connections, are learning to write, code, and process information by ingesting vast amounts of data—essentially a large fraction of the internet.

This scaling, however, demands enormous computing power. According to nonprofit AI research organization Epoch AI, the computing power dedicated to AI training has been quadrupling every year. Should this trend continue, by 2030, AI models could be trained with up to 10,000 times more compute than today’s cutting-edge models, such as OpenAI’s GPT-4. This would represent a leap as significant as the difference between GPT-2’s rudimentary text generation in 2019 and GPT-4’s advanced problem-solving abilities in 2023.

However, modern AI systems already consume significant amounts of energy, rely on tens of thousands of specialized chips, and require access to trillions of online data points. With chip shortages and concerns about running out of quality data looming, it begs the question: Is this level of growth sustainable or even technically feasible?

In its latest report, Epoch AI identified four major constraints that could limit AI’s ability to scale: power, chips, data, and latency. While the growth is technically possible, the road ahead is filled with challenges.

1. Power: The Biggest Bottleneck AI scaling requires enormous amounts of energy. For example, Meta’s latest AI model was trained using 16,000 of Nvidia’s most powerful chips, consuming 27 megawatts of electricity—the equivalent of what 23,000 U.S. households use annually. By 2030, training frontier AI models could require up to 200 times more power, amounting to about 6 gigawatts, or 30% of all power consumed by data centers today.While it’s possible for companies to tap into multiple power plants and distribute training across various data centers to share the load, this approach would require high-speed fiber connections and infrastructure expansion. Despite the challenges, it’s technically feasible, with potential power consumption estimates ranging from 1 gigawatt to 45 gigawatts, depending on how data centers distribute their power demands.
2. Chips: The AI Engine AI models are trained using graphics processing units (GPUs), with Nvidia leading the market in their production. While there’s enough capacity to produce GPUs, the bottleneck lies in high-bandwidth memory and chip packaging. Forecasts suggest that by 2030, AI labs could have access to anywhere between 20 and 400 million chips, allowing for up to 50,000 times more computing power for AI training than is available today.
3. Data: A Potential Scarcity Data is the fuel that powers AI models, but high-quality publicly available data may run dry by 2026. Epoch notes that while AI models will still have enough data to scale through 2030, concerns remain over copyright lawsuits and the supply of reliable information. However, AI labs are expanding beyond text, training models with image, audio, and video data, which could alleviate some of the pressure.Synthetic data, generated artificially for training purposes, also presents a potential solution. Companies like DeepMind and Meta have already employed synthetic data in their models. Yet, over-reliance on this approach may degrade model quality and would require even more computing power to generate.
4. Latency: Speed of Scaling As AI models grow larger, the time it takes for data to be processed within and between data centers becomes a limiting factor. While today’s infrastructure allows for a significant increase in model size, this will eventually reach a practical ceiling. Nonetheless, Epoch estimates that AI models could still be scaled up to 1,000,000 times the compute power of GPT-4 before hitting latency bottlenecks.

The biggest assumption in all this is the continued growth of AI investment. As scaling continues, so does the price tag. Current models may cost up to $1 billion to train, and the next generation could cost as much as $10 billion or more. Microsoft, for instance, has already committed to investing that amount in its Stargate AI supercomputer, slated for completion in 2028.

Yet, there’s no guarantee that this level of investment will continue indefinitely. If future AI models don’t deliver meaningful improvements or the market fails to justify the rising costs, investors may begin to question whether scaling is worth the price. Moreover, critics argue that large AI models may hit a point of diminishing returns, with limited additional value from further scaling.

If AI scaling continues uninterrupted, the impact on the economy could be profound, with some estimating returns in the trillions of dollars. However, whether AI labs can sustain this growth depends on overcoming key challenges in power, chips, data, and latency. As companies like Alphabet and Microsoft pour billions into AI, the future of this rapidly evolving technology hangs on its ability to meet ever-increasing demands.

In the end, the question remains: Will the exponential growth of AI lead to revolutionary breakthroughs, or will it encounter insurmountable roadblocks? The next decade will likely hold the answer.

By Impact Lab