The remarkable progress in artificial intelligence (AI) over recent years can largely be attributed to one factor: scale. Around the beginning of this decade, researchers noticed that as they increased the size of their algorithms and fed them more data, the capabilities of AI models skyrocketed. The latest AI models, boasting hundreds of billions to over a trillion network connections, are learning to write, code, and process information by ingesting vast amounts of data—essentially a large fraction of the internet.
This scaling, however, demands enormous computing power. According to nonprofit AI research organization Epoch AI, the computing power dedicated to AI training has been quadrupling every year. Should this trend continue, by 2030, AI models could be trained with up to 10,000 times more compute than today’s cutting-edge models, such as OpenAI’s GPT-4. This would represent a leap as significant as the difference between GPT-2’s rudimentary text generation in 2019 and GPT-4’s advanced problem-solving abilities in 2023.
Continue reading… “Scaling AI: The Road to a Technological Breakthrough or Bottleneck?”
