我们的使命是确保 artificial general intelligence(AGI,通用人工智能)——在总体上比人类更聪明的 AI 系统——造福全人类(https://openai.com/charter/)。 Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity (https://openai.com/charter/). ¶
如果 AGI 被成功创造出来,这项技术可以帮助提升人类:扩大丰裕、给全球经济装上涡轮增压,并助力发现能够改写可能性边界的新科学知识。 If AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging the global economy, and aiding in the discovery of new scientific knowledge that changes the limits of possibility. ¶
AGI 有潜力赋予每个人不可思议的新能力;我们可以想象这样一个世界:所有人都能在几乎任何认知任务上获得帮助,这将成为人类才智与创造力的巨大倍增器。 AGI has the potential to give everyone incredible new capabilities; we can imagine a world where all of us have access to help with almost any cognitive task, providing a great force multiplier for human ingenuity and creativity. ¶
另一方面,AGI 也伴随着被滥用的严重风险、重大事故风险和社会动荡风险。由于 AGI 的正面潜力如此巨大,我们不认为社会能够、也不应该永远停止其发展;相反,社会和 AGI 的开发者必须想办法把它做对。[^gifts] On the other hand, AGI would also come with serious risk of misuse, drastic accidents, and societal disruption. Because the upside of AGI is so great, we do not believe it is possible or desirable for society to stop its development forever; instead, society and the developers of AGI have to figure out how to get it right.[^gifts] ¶
虽然我们无法确切预测将会发生什么——当然,我们当前的进展也有可能撞墙——但我们可以阐明自己最在意的原则: Although we cannot predict exactly what will happen, and of course our current progress could hit a wall, we can articulate the principles we care about most: ¶
- 我们希望 AGI 赋能人类,让人类在宇宙中最大限度地繁盛。我们并不预期未来会是一个毫无瑕疵的乌托邦,但我们希望最大化好的结果、最小化坏的结果,并让 AGI 成为人类的放大器。 - We want AGI to empower humanity to maximally flourish in the universe. We don’t expect the future to be an unqualified utopia, but we want to maximize the good and minimize the bad, and for AGI to be an amplifier of humanity. ¶
- 我们希望 AGI 的收益、使用权和治理权得到广泛而公平的分享。 - We want the benefits of, access to, and governance of AGI to be widely and fairly shared. ¶
- 我们希望成功驾驭巨大的风险。面对这些风险,我们承认:理论上看似正确的东西,实践中往往比预期更怪异。我们相信,必须通过部署能力较弱版本的技术来不断学习和适应,从而尽量避免「只有一次机会把它做对」的局面。 - We want to successfully navigate massive risks. In confronting these risks, we acknowledge that what seems right in theory often plays out more strangely than expected in practice. We believe we have to continuously learn and adapt by deploying less powerful versions of the technology in order to minimize “one shot to get it right” scenarios. ¶
## 短期 The short term ¶
有几件事,我们认为现在就该做,以便为 AGI 做好准备。 There are several things we think are important to do now to prepare for AGI. ¶
第一,随着我们创造出一代代更强大的系统,我们希望将它们部署出去,积累在真实世界中运行它们的经验。我们相信,这是审慎引导 AGI 降生的最佳方式——渐进地过渡到拥有 AGI 的世界,好于一蹴而就。我们预期强大的 AI 会让世界的进步速度大大加快,而我们认为渐进地适应这种加速更好。 First, as we create successively more powerful systems, we want to deploy them and gain experience with operating them in the real world. We believe this is the best way to carefully steward AGI into existence—a gradual transition to a world with AGI is better than a sudden one. We expect powerful AI to make the rate of progress in the world much faster, and we think it’s better to adjust to this incrementally. ¶
渐进的过渡能给人、政策制定者和机构留出时间去理解正在发生的事情,亲身感受这些系统的益处与弊端,调整我们的经济,并把监管落实到位。它还允许社会与 AI 共同演化,让人们在风险相对较低的时候,集体想清楚自己究竟想要什么。 A gradual transition gives people, policymakers, and institutions time to understand what’s happening, personally experience the benefits and downsides of these systems, adapt our economy, and to put regulation in place. It also allows for society and AI to co-evolve, and for people collectively to figure out what they want while the stakes are relatively low. ¶
我们目前认为,成功驾驭 AI 部署挑战的最佳方式,是「快速学习 + 谨慎迭代」的紧密反馈循环。社会将面临一系列重大问题:AI 系统可以被允许做什么、如何对抗偏见、如何应对工作被取代,等等。最优决策将取决于技术所走的路径;和任何新领域一样,迄今大多数专家的预测都是错的。这使得在真空中做规划非常困难。[^planning] We currently believe the best way to successfully navigate AI deployment challenges is with a tight feedback loop of rapid learning and careful iteration. Society will face major questions about what AI systems are allowed to do, how to combat bias, how to deal with job displacement, and more. The optimal decisions will depend on the path the technology takes, and like any new field, most expert predictions have been wrong so far. This makes planning in a vacuum very difficult.[^planning] ¶
笼统地说,我们认为世界上更多的 AI 使用将带来好处,并希望促进它(比如把模型放进我们的 API、将它们开源等等)。我们相信,民主化的使用权还会带来更多更好的研究、权力的去中心化、更多的收益,以及更广泛的人群贡献新想法。 Generally speaking, we think more usage of AI in the world will lead to good, and want to promote it (by putting models in our API, open-sourcing them, etc.). We believe that democratized access will also lead to more and better research, decentralized power, more benefits, and a broader set of people contributing new ideas. ¶
随着我们的系统越来越接近 AGI,我们在模型的创建和部署上正变得越来越谨慎。我们的决策需要比社会通常对待新技术时谨慎得多,也比许多用户所希望的更谨慎。AI 领域里有些人认为 AGI(及其后继系统)的风险是虚构的;如果他们最终被证明是对的,我们会非常高兴,但我们将按「这些风险关乎存亡」的假设来运作(https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)。 As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models. Our decisions will require much more caution than society usually applies to new technologies, and more caution than many users would like. Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks are existential (https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/). ¶
到了某个时点,部署的正面与负面(例如赋能恶意行为者、造成社会与经济动荡、加速不安全的竞赛)之间的平衡可能发生转变,届时我们将大幅改变持续部署的计划。 At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans around continuous deployment. ¶
随着我们的系统越来越接近 AGI,我们在模型的创建和部署上正变得越来越谨慎。 As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models. ¶
第二,我们正致力于创建对齐程度越来越高、可引导性越来越强的模型。我们从第一版 GPT-3 这类模型转向 InstructGPT(https://openai.com/blog/instruction-following/)和 ChatGPT(https://chat.openai.com/),就是这方面的一个早期例子。 Second, we are working towards creating increasingly aligned and steerable models. Our shift from models like the first version of GPT-3 to InstructGPT (https://openai.com/blog/instruction-following/) and ChatGPT (https://chat.openai.com/) is an early example of this. ¶
我们尤其认为重要的是:社会应在「AI 可以被如何使用」上约定极为宽泛的边界,而在这些边界之内,个体用户拥有很大的自主裁量。我们的最终愿望是世界各国的机构共同商定这些宽泛边界应划在哪里;短期内我们计划开展引入外部意见的实验。世界各国机构还需要补强能力与经验,为围绕 AGI 的复杂决策做好准备。 In particular, we think it’s important that society agree on extremely wide bounds of how AI can be used, but that within those bounds, individual users have a lot of discretion. Our eventual hope is that the institutions of the world agree on what these wide bounds should be; in the shorter term we plan to run experiments for external input. The institutions of the world will need to be strengthened with additional capabilities and experience to be prepared for complex decisions about AGI. ¶
我们产品的「默认设置」很可能会受到相当的限制,但我们计划让用户能方便地改变自己所使用 AI 的行为。我们相信应当赋能个人自主决策,也相信思想多样性所蕴含的力量。 The “default setting” of our products will likely be quite constrained, but we plan to make it easy for users to change the behavior of the AI they’re using. We believe in empowering individuals to make their own decisions and the inherent power of diversity of ideas. ¶
随着模型变得更强大,我们需要开发新的对齐技术(https://openai.com/blog/our-approach-to-alignment-research/)(以及用以了解现有技术何时失效的测试)。我们短期的计划是用 AI 帮助人类评估(https://openai.com/blog/critiques/)更复杂模型的输出、监控复杂系统;长期计划是用 AI 帮助我们想出更好的对齐技术的新思路。 We will need to develop new alignment techniques (https://openai.com/blog/our-approach-to-alignment-research/) as our models become more powerful (and tests to understand when our current techniques are failing). Our plan in the shorter term is to use AI to help humans evaluate (https://openai.com/blog/critiques/) the outputs of more complex models and monitor complex systems, and in the longer term to use AI to help us come up with new ideas for better alignment techniques. ¶
重要的是,我们认为 AI 安全与能力往往必须一起进步。把两者分开谈论是一个伪二分法;它们在许多方面相互关联。我们最好的安全工作都出自与最强模型的协同攻关。尽管如此,安全进步与能力进步之比必须不断提高,这一点很重要。 Importantly, we think we often have to make progress on AI safety and capabilities together. It’s a false dichotomy to talk about them separately; they are correlated in many ways. Our best safety work has come from working with our most capable models. That said, it’s important that the ratio of safety progress to capability progress increases. ¶
第三,我们期待一场围绕三个关键问题的全球对话:如何治理这些系统、如何公平分配它们产生的收益、如何公平分享使用权。 Third, we hope for a global conversation about three key questions: how to govern these systems, how to fairly distribute the benefits they generate, and how to fairly share access. ¶
除了这三个方面,我们还尝试建立一种让我们的激励与好结果对齐的架构。我们的宪章(https://openai.com/charter/)中有一条规定:在 AGI 开发的后期阶段,协助其他组织推进安全,而不是与它们竞赛。我们为股东可获得的回报设置了上限,这样我们就不会有动机去无节制地攫取价值、冒着部署潜在灾难性危险系统的风险行事(当然,这也是与社会分享收益的一种方式)。我们由一个非营利组织治理,它让我们为人类利益而运作(并且可以否决任何营利性利益),包括让我们能够在安全需要时取消对股东的股权义务,并资助了世界上最全面的 UBI 实验。 In addition to these three areas, we have attempted to set up our structure in a way that aligns our incentives with a good outcome. We have a clause in our Charter (https://openai.com/charter/) about assisting other organizations to advance safety instead of racing with them in late-stage AGI development. We have a cap on the returns our shareholders can earn so that we aren’t incentivized to attempt to capture value without bound and risk deploying something potentially catastrophically dangerous (and of course as a way to share the benefits with society). We have a nonprofit that governs us and lets us operate for the good of humanity (and can override any for-profit interests), including letting us do things like cancel our equity obligations to shareholders if needed for safety and sponsor the world’s most comprehensive UBI experiment. ¶
我们尝试建立一种让我们的激励与好结果对齐的架构。 We have attempted to set up our structure in a way that aligns our incentives with a good outcome. ¶
我们认为,像我们这样的努力在发布新系统之前接受独立审计是重要的;今年晚些时候我们会详谈这一点。到了某个时点,在开始训练未来系统之前获得独立审查可能会变得重要,而对于最前沿的努力而言,同意对用于创建新模型的算力增长率加以限制也可能会变得重要。我们认为,就「一项 AGI 努力何时应停止一次训练、何时认定一个模型可以安全发布、何时将一个模型撤出生产使用」建立公共标准是重要的。最后,我们认为,让主要世界大国政府对超过一定规模的训练有知情权是重要的。 We think it’s important that efforts like ours submit to independent audits before releasing new systems; we will talk about this in more detail later this year. At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models. We think public standards about when an AGI effort should stop a training run, decide a model is safe to release, or pull a model from production use are important. Finally, we think it’s important that major world governments have insight about training runs above a certain scale. ¶
## 长期 The long term ¶
我们相信,人类的未来应由人类决定,而且与公众分享进展信息是重要的。所有试图构建 AGI 的努力都应接受严格审视,重大决策都应征求公众意见。 We believe that the future of humanity should be determined by humanity, and that it’s important to share information about progress with the public. There should be great scrutiny of all efforts attempting to build AGI and public consultation for major decisions. ¶
第一个 AGI 将只是智能连续谱上的一个点。我们认为进展很可能会从那里继续下去,并可能在很长时期内维持过去十年我们所见的进步速度。如果果真如此,世界可能变得与今天截然不同,风险也可能超乎寻常。一个未对齐的超级智能 AGI 可能给世界造成严重伤害;一个在超级智能上拥有决定性领先优势的专制政权同样可能。 The first AGI will be just a point along the continuum of intelligence. We think it’s likely that progress will continue from there, possibly sustaining the rate of progress we’ve seen over the past decade for a long period of time. If this is true, the world could become extremely different from how it is today, and the risks could be extraordinary. A misaligned superintelligent AGI could cause grievous harm to the world; an autocratic regime with a decisive superintelligence lead could do that too. ¶
能够加速科学的 AI 是一个值得特别思考的特殊案例,而且可能比其他一切都更具影响力。能力足以加速自身进步的 AGI,有可能让重大变化快得出人意料地发生(即使转变开始时很缓慢,我们也预期它在最后阶段会相当快)。我们认为较慢的起飞(takeoff)更容易保证安全,而各 AGI 努力之间在关键节点上协调减速很可能至关重要(即使在一个我们不需要靠减速来解决技术对齐问题的世界里,减速对于给社会留出足够的适应时间可能仍然重要)。 AI that can accelerate science is a special case worth thinking about, and perhaps more impactful than everything else. It’s possible that AGI capable enough to accelerate its own progress could cause major changes to happen surprisingly quickly (and even if the transition starts slowly, we expect it to happen pretty quickly in the final stages). We think a slower takeoff is easier to make safe, and coordination among AGI efforts to slow down at critical junctures will likely be important (even in a world where we don’t need to do this to solve technical alignment problems, slowing down may be important to give society enough time to adapt). ¶
成功过渡到一个拥有超级智能的世界,也许是人类历史上最重要——也最充满希望、最令人恐惧——的工程。成功远非必然,而其赌注(无上限的下行与无上限的上行)有望把我们所有人团结起来。 Successfully transitioning to a world with superintelligence is perhaps the most important—and hopeful, and scary—project in human history. Success is far from guaranteed, and the stakes (boundless downside and boundless upside) will hopefully unite all of us. ¶
我们可以想象一个人类繁盛到我们任何人都还无法完全想象的程度的世界。我们希望为世界贡献一个与这种繁盛相对齐的 AGI。 We can imagine a world in which humanity flourishes to a degree that is probably impossible for any of us to fully visualize yet. We hope to contribute to the world an AGI aligned with such flourishing. ¶