Scaling up: how increasing inputs has made artificial intelligence more capable
本篇导读
过去十年 AI 的进步,主要不是靠"灵光一闪"的新算法,而是靠"堆规模":更多数据、更多参数、更多算力。这篇文章用三条指数曲线把这件事讲清楚,并解释了为什么这种增长让人既兴奋又不安。
语言平实、数字多、比喻多(旋钮、鼠标走迷宫、显卡算力换算),非常适合练习"读数据类科普"时的抓重点能力。
一、英文正文
For most of Artificial Intelligence’s (AI’s) history, many researchers expected that building truly capable systems would need a long series of scientific breakthroughs: revolutionary algorithms, deep insights into human cognition, or fundamental advances in our understanding of the brain. While scientific advances have played a role, recent AI progress has revealed an unexpected insight: a lot of the recent improvement in AI capabilities has come simply from scaling up existing AI systems.
Here, scaling means deploying more computational power, using larger datasets, and building bigger models. This approach has worked surprisingly well so far. Just a few years ago, state-of-the-art AI systems struggled with basic tasks like counting. Today, they can solve complex math problems, write software, create extremely realistic images and videos, and discuss academic topics.
[...]
What is scaling in AI models?
Let’s briefly break down what scaling means in AI. Scaling is about increasing three main things during training, which typically need to grow together:
[...]
The idea is simple but powerful: bigger AI systems, trained on more data and using more computational resources, tend to perform better. Even without substantial changes to the algorithms, this approach often leads to better performance across many tasks.
Here is another reason why this is important: as researchers scale up these AI systems, they not only improve in the tasks they were trained on but can sometimes lead them to develop new abilities that they did not have on a smaller scale. For example, language models initially struggled with simple arithmetic tests like three-digit addition, but larger models could handle these easily once they reached a certain size. The transition wasn't a smooth, incremental improvement but a more abrupt leap in capabilities.
This abrupt jump in capability, rather than steady improvement, can be concerning. If, for example, models suddenly develop unexpected and potentially harmful behaviors simply as a result of getting bigger, it would be harder to anticipate and control.
This makes tracking these metrics important.
What are the three components of scaling up AI models?
Data: scaling up the training data
One way to view today's AI models is by looking at them as very sophisticated pattern recognition systems. They work by identifying and learning from statistical regularities in the text, images, or other data on which they are trained. The more data the model has access to, the more it can learn about the nuances and complexities of the knowledge domain in which it’s designed to operate.
In 1950, Claude Shannon built one of the earliest examples of “AI”: a robotic mouse named Theseus that could "remember" its path through a maze using simple relay circuits. Each wall Theseus bumped into became a data point, allowing it to learn the correct route. The total number of walls or data points was 40. You can find this data point in the chart; it is the first one.
[...]
All recent notable AI models — especially large, state-of-the-art ones — rely on vast amounts of training data. With the y-axis displayed on a logarithmic scale, the chart shows that the data used to train AI models has grown exponentially. From 40 data points for Theseus to trillions of data points for the largest modern systems in a little more than seven decades.
Since 2010, the training data has doubled approximately every nine to ten months. You can see this rapid growth in the chart, shown by the purple line extending from the start of 2010 to October 2024, the latest data point as I write this article.
Datasets used for training large language models, in particular, have experienced an even faster growth rate, tripling in size each year since 2010. Large language models process text by breaking it into tokens — basic units the model can encode and understand. A token doesn't directly correspond to one word, but on average, three English words correspond to about four tokens.
GPT-2, released in 2019, is estimated to have been trained on 4 billion tokens, roughly equivalent to 3 billion words. To put this in perspective, as of September 2024, the English Wikipedia contained around 4.6 billion words. In comparison, GPT-4, released in 2023, was trained on almost 13 trillion tokens, or about 9.75 trillion words. This means that GPT-4’s training data was equivalent to over 2000 times the amount of text of the entire English Wikipedia.
As we use more data to train AI systems, we might eventually run out of high-quality human-generated materials like books, articles, and research papers. Some researchers predict we could exhaust useful training materials within the next few decades. While AI models themselves can generate vast amounts of data, training AI on machine-generated materials could create problems, making the models less accurate and more repetitive.
Parameters: scaling up the model size
Increasing the amount of training data lets AI models learn from much more information than ever before. However, to pick up on the patterns in this data and learn effectively, models need what are called "parameters". Parameters are a bit like knobs that can be tweaked to improve how the model processes information and makes predictions. As the amount of training data grows, models need more capacity to capture all the details in the training data. This means larger datasets typically require the models to have more parameters to learn effectively.
Early neural networks had hundreds or thousands of parameters. [...] Recent large models, such as GPT-3, boast up to 175 billion parameters. While the raw number may seem large, this roughly translates into 700 GB if stored on a disc, which is easily manageable by today’s computers.
[...]
While bigger AI models can do more, they also face some problems. One major issue is called "overfitting". This happens when an AI becomes “too optimized” for processing the particular data it was trained on but struggles with new data. To combat this, researchers employ two strategies: implementing specialized techniques for more generalized learning and expanding the volume and diversity of training data.
Compute: scaling up computational resources
As AI models grow in data and parameters, they require exponentially more computational resources. These resources, commonly referred to as “compute” in AI research, are typically measured in total floating-point operations (“FLOP”), where each FLOP represents a single arithmetic calculation like addition or multiplication.
[...]
[...] However, since 2010, this growth has accelerated dramatically, now doubling approximately every six months, with the most compute-intensive model reaching 50 billion petaFLOP as I write this article.
To put this scale in perspective, a single high-end graphics card like the NVIDIA GeForce RTX 3090 — widely used in AI research — running at full capacity for an entire year would complete just 1.1 million petaFLOP computations. 50 billion petaFLOP is approximately 45,455 times more than that.
Compute, data, and parameters tend to scale at the same time
Compute, data, and parameters are closely interconnected when it comes to scaling AI models. When AI models are trained on more data, there are more things to learn. To deal with the increasing complexity of the data, AI models, therefore, require more parameters to learn from the various features of the data. Adding more parameters to the model means that it needs more computational resources during training.
This interdependence means that data, parameters, and compute need to grow simultaneously. [...]
What can we learn from these trends for the future of AI?
Companies are seeking large financial investments to develop and scale their AI models, with a growing focus on generative AI technologies. At the same time, the key hardware that is used for training — GPUs — is getting much cheaper and more powerful, with its computing speed doubling roughly every 2.5 years per dollar spent. Some organizations are also now leveraging more computational resources not just in training AI models but also during inference — the phase when models generate responses — as illustrated by OpenAI's latest o1 model.
These developments could help create more sophisticated AI technologies faster and cheaper. As companies invest more money and the necessary hardware improves, we might see significant improvements in what AI can do, including potentially unexpected new capabilities.
Because these changes could have major effects on our society, it's important that we track and understand these developments early on. To support this, we will update key metrics — such as the growth in computational resources, training data volumes, and model parameters — on a monthly basis. These updates will help monitor the rapid evolution of AI technologies and provide valuable insights into their trajectory.
二、生词与短语精讲
用法
先通读正文,再回来核对。建议先遮住中文释义自测。
| 单词/短语 | 音标 | 词性 | 英文释义 | 中文释义 | 文中例句 |
|---|---|---|---|---|---|
| breakthrough | /ˈbreɪkθruː/ | n. | an important discovery that removes a barrier | 突破、重大进展 | Many researchers expected that building truly capable systems would need a long series of scientific breakthroughs: revolutionary algorithms, deep insights into human cognition, or fundamental advances in our understanding of the brain. ... |
| scale up | /skeɪl ʌp/ | phrasal v. | to increase the size, amount or power of something | 扩大规模、增大 | A lot of the recent improvement in AI capabilities has come simply from scaling up existing AI systems. ... |
| state-of-the-art | /ˌsteɪt əv ði ˈɑːt/ | adj. | the most advanced and modern of its kind | 最先进的 | Just a few years ago, state-of-the-art AI systems struggled with basic tasks like counting. |
| incremental | /ˌɪŋkrəˈmentl/ | adj. | happening gradually in small steps | 渐进的、逐步的 | The transition wasn't a smooth, incremental improvement but a more abrupt leap in capabilities. |
| abrupt | /əˈbrʌpt/ | adj. | sudden and unexpected | 突然的、突兀的 | The transition wasn't a smooth, incremental improvement but a more abrupt leap in capabilities. |
| anticipate | /ænˈtɪsɪpeɪt/ | v. | to expect something and prepare for it | 预判、预料 | If, for example, models suddenly develop unexpected and potentially harmful behaviors simply as a result of getting bigger, it would be harder to anticipate and control. |
| sophisticated | /səˈfɪstɪkeɪtɪd/ | adj. | complex, advanced and well developed | 复杂精密的、高级的 | One way to view today's AI models is by looking at them as very sophisticated pattern recognition systems. ... |
| regularity | /ˌreɡjuˈlærəti/ | n. | a pattern that appears repeatedly | 规律性 | They work by identifying and learning from statistical regularities in the text, images, or other data on which they are trained. |
| nuance | /ˈnjuːɑːns/ | n. | a very slight difference in meaning or detail | 细微差别 | The more data the model has access to, the more it can learn about the nuances and complexities of the knowledge domain in which it’s designed to operate. |
| maze | /meɪz/ | n. | a complicated network of paths that is hard to get through | 迷宫 | In 1950, Claude Shannon built one of the earliest examples of “AI”: a robotic mouse named Theseus that could "remember" its path through a maze using simple relay circuits. |
| logarithmic | /ˌlɒɡəˈrɪðmɪk/ | adj. | using a scale in which equal steps represent multiples | 对数的 | With the y-axis displayed on a logarithmic scale, the chart shows that the data used to train AI models has grown exponentially. |
| exponential | /ˌekspəˈnenʃl/ | adj. | growing extremely fast, at an ever-increasing rate | 指数式的 | The data used to train AI models has grown exponentially. ... |
| triple | /ˈtrɪpl/ | v. | to become three times as large | 增至三倍 | Datasets used for training large language models, in particular, have experienced an even faster growth rate, tripling in size each year since 2010. |
| equivalent | /ɪˈkwɪvələnt/ | adj. | equal in value, amount or meaning | 等同的、相当的 | GPT-2, released in 2019, is estimated to have been trained on 4 billion tokens, roughly equivalent to 3 billion words. |
| exhaust | /ɪɡˈzɔːst/ | v. | to use something up completely | 耗尽、用完 | Some researchers predict we could exhaust useful training materials within the next few decades. |
| repetitive | /rɪˈpetətɪv/ | adj. | doing or saying the same thing many times | 重复的 | Training AI on machine-generated materials could create problems, making the models less accurate and more repetitive. ... |
| tweak | /twiːk/ | v. | to make small adjustments to improve something | 微调 | Parameters are a bit like knobs that can be tweaked to improve how the model processes information and makes predictions. |
| overfitting | /ˌəʊvəˈfɪtɪŋ/ | n. | when a model fits its training data so closely that it fails on new data | 过拟合 | One major issue is called "overfitting". |
| combat | /ˈkɒmbæt/ | v. | to try to stop or reduce something bad | 应对、抑制 | To combat this, researchers employ two strategies: implementing specialized techniques for more generalized learning and expanding the volume and diversity of training data. ... |
| accelerate | /əkˈseləreɪt/ | v. | to happen faster or make something happen faster | 加速 | Since 2010, this growth has accelerated dramatically, now doubling approximately every six months, with the most compute-intensive model reaching 50 billion petaFLOP as I write this article. ... |
| interconnected | /ˌɪntəkəˈnektɪd/ | adj. | linked to each other so that one affects the other | 相互关联的 | Compute, data, and parameters are closely interconnected when it comes to scaling AI models. ... |
| simultaneously | /ˌsɪmlˈteɪniəsli/ | adv. | at the same time | 同时地 | Data, parameters, and compute need to grow simultaneously. ... |
| leverage | /ˈliːvərɪdʒ/ | v. | to use something you have in order to achieve a goal | 利用、借助 | Some organizations are also now leveraging more computational resources not just in training AI models but also during inference — the phase when models generate responses — as illustrated by OpenAI's latest o1 model. |
| inference | /ˈɪnfərəns/ | n. | the stage when a trained model produces an answer | 推理(模型输出阶段) | Some organizations are also now leveraging more computational resources not just in training AI models but also during inference — the phase when models generate responses — as illustrated by OpenAI's latest o1 model. |
| trajectory | /trəˈdʒektəri/ | n. | the path that something follows as it develops | 发展轨迹 | These updates will help monitor the rapid evolution of AI technologies and provide valuable insights into their trajectory. ... |
高频搭配与短语动词
| 搭配 | 中文 | 例句 |
|---|---|---|
| play a role | 起作用 | While scientific advances have played a role, recent AI progress has revealed an unexpected insight: a lot of the recent improvement in AI capabilities has come simply from scaling up existing AI systems. |
| break down | 拆解、分解说明 | Let’s briefly break down what scaling means in AI. |
| lead to | 导致、带来 | Even without substantial changes to the algorithms, this approach often leads to better performance across many tasks. |
| run out of | 用完、耗尽 | As we use more data to train AI systems, we might eventually run out of high-quality human-generated materials like books, articles, and research papers. |
| to put this in perspective | 为了便于比较、换个角度看 | To put this in perspective, as of September 2024, the English Wikipedia contained around 4.6 billion words. |
| pick up on | 注意到、捕捉到 | However, to pick up on the patterns in this data and learn effectively, models need what are called "parameters". |
| a bit like | 有点像 | Parameters are a bit like knobs that can be tweaked to improve how the model processes information and makes predictions. |
| at full capacity | 满负荷地 | To put this scale in perspective, a single high-end graphics card like the NVIDIA GeForce RTX 3090 — widely used in AI research — running at full capacity for an entire year would complete just 1.1 million petaFLOP computations. |
| when it comes to | 在……方面 | Compute, data, and parameters are closely interconnected when it comes to scaling AI models. |
| be referred to as | 被称为 | These resources, commonly referred to as “compute” in AI research, are typically measured in total floating-point operations (“FLOP”), where each FLOP represents a single arithmetic calculation like addition or multiplication. |
三、长难句解析
长难句 1
原句:For most of Artificial Intelligence’s (AI’s) history, many researchers expected that building truly capable systems would need a long series of scientific breakthroughs: revolutionary algorithms, deep insights into human cognition, or fundamental advances in our understanding of the brain.
结构拆解:主句主干是 many researchers expected that...,For most of AI’s history 是时间状语。that 引导宾语从句,从句主语是动名词短语 building truly capable systems,谓语是 would need,宾语是 a long series of scientific breakthroughs。冒号后是三个并列的名词短语,对 breakthroughs 作同位式列举:revolutionary algorithms、deep insights into human cognition、fundamental advances in our understanding of the brain。
参考翻译:在人工智能发展史的绝大部分时间里,许多研究者认为,要造出真正有能力的系统,需要一系列接连不断的科学突破:革命性的算法、对人类认知的深刻洞见,或者对我们如何理解大脑的根本性推进。
长难句 2
原句:Here is another reason why this is important: as researchers scale up these AI systems, they not only improve in the tasks they were trained on but can sometimes lead them to develop new abilities that they did not have on a smaller scale.
结构拆解:冒号前是主句 Here is another reason,why this is important 是定语从句修饰 reason。冒号后 as researchers scale up these AI systems 是时间状语从句;主句用 not only... but (also)... 连接两个并列谓语 improve 与 can lead,they were trained on 是省略了 that 的定语从句修饰 the tasks;lead them to develop new abilities 是"动词+宾语+不定式"结构;句末 that they did not have on a smaller scale 又是定语从句修饰 abilities。
参考翻译:这里还有一个它为什么重要的理由:当研究者把这些 AI 系统做大时,它们不仅在训练任务上表现更好,有时还会发展出在较小规模时不具备的新能力。
长难句 3
原句:To put this scale in perspective, a single high-end graphics card like the NVIDIA GeForce RTX 3090 — widely used in AI research — running at full capacity for an entire year would complete just 1.1 million petaFLOP computations.
结构拆解:句首 To put this scale in perspective 是不定式短语作目的状语,意为"为了让人看清这个量级"。主句主干是 a graphics card would complete just 1.1 million petaFLOP computations,主语被两处修饰拉开:like the NVIDIA GeForce RTX 3090 是介词短语作后置定语;两个破折号之间的 widely used in AI research 是过去分词短语作插入语,补充说明这张显卡的应用场景。running at full capacity for an entire year 是现在分词短语作后置定语,相当于 which runs...,说明计算的前提条件。
参考翻译:为了对这样的量级有个直观感受:像英伟达 GeForce RTX 3090 这样一张在 AI 研究中被广泛使用的高端显卡,满负荷运行整整一年,也只完成约 110 万 petaFLOP 的计算量。
四、专业词汇标注
专业词汇
这些词只需"认识",不必会拼写。
| 专业词汇 | 中文解释 | 所属领域 |
|---|---|---|
| scaling | 规模扩张:同时增大数据、参数与算力 | 机器学习 |
| parameters | 参数:模型中可被调整的数值,"旋钮" | 机器学习 |
| compute | 算力:训练或运行模型所需的计算资源总量 | 人工智能工程 |
| FLOP / petaFLOP | 浮点运算次数/千万亿次浮点运算,衡量算力的单位 | 计算机体系结构 |
| token | 词元:模型处理文本的基本单位,约 3 个英文单词对应 4 个词元 | 自然语言处理 |
| neural network | 神经网络:由大量可调参数构成的模型结构 | 机器学习 |
| overfitting | 过拟合:对训练数据过度贴合,遇到新数据就失灵 | 机器学习 |
| training data | 训练数据:模型学习规律所用的文本、图像等材料 | 机器学习 |
| large language model (LLM) | 大语言模型,如 GPT 系列 | 自然语言处理 |
| inference | 推理:模型训练完成后生成回答的阶段 | 人工智能工程 |
| GPU | 图形处理器,AI 训练的主力硬件 | 计算机硬件 |
| logarithmic scale | 对数坐标:等距刻度表示倍数关系,用于展示指数增长 | 数据可视化 |
五、阅读训练建议
- [ ] 第一遍:不看注释通读全文,记录用时 ___ 分钟
- [ ] 第二遍:查生词表,回读卡住的句子
- [ ] 第三遍:只看英文,口头复述每段大意
- [ ] 输出:用 5 句英文写出本文摘要
关联阅读
- 01-01-算法单一化对招聘的影响 —— 模型越准,用它的机构越多,社会后果就越集中:本篇解释了"准确率"从何而来。
- 01-02-人工智能风险管理框架 —— 规模扩张带来的"能力突跳",正是框架里"难以预料的失败"的现实注脚。
- 01-04-大语言模型对医疗意味着什么 —— 把大语言模型放进诊室后,准确率与可解释性会遇到哪些具体麻烦。
六、来源与版权说明
- 原文标题:Scaling up: how increasing inputs has made artificial intelligence more capable
- 来源:Our World in Data,2025 年 1 月 20 日
- 原文链接:https://ourworldindata.org/scaling-up-ai
- 说明:本篇英文正文为原文节选(删节处标
[...]),仅用于个人英语学习;版权归原作者与发布机构所有(原文采用 CC BY 许可)。