这是一个两篇系列文章的第二篇——第一篇在这里(http://blog.samaltman.com/machine-intelligence-part-1)。 This is part two of a a two-part post—the first part is here (http://blog.samaltman.com/machine-intelligence-part-1). ¶
监管的必要性 THE NEED FOR REGULATION ¶
尽管最近关于机器智能危险的讨论很多,但对于我们应该做些什么来缓解这一威胁,讨论却很少。 Although there has been a lot of discussion about the dangers of machine intelligence recently, there hasn’t been much discussion about what we should try to do to mitigate the threat. ¶
部分原因在于,许多人几乎为自己如此坚信「其神经元中的算法永远不会在硅片上复现」而感到自豪,所以他们不认为这是一个潜在威胁。另一部分原因在于,弄清该拿它怎么办实在太难了,而且越是深入思考,就越觉得不可能。还有一部分原因是,超人类机器智能(superhuman machine intelligence,SMI)大概还要几十年才会出现 [1],而我们眼下就有非常紧迫的问题要处理。 Part of the reason is that many people are almost proud of how strongly they believe that the algorithms in their neurons will never be replicated in silicon, and so they don’t believe it’s a potential threat. Another part of it is that figuring out what to do about it is just very hard, and the more one thinks about it the less possible it seems. And another part is that superhuman machine intelligence (SMI) is probably still decades away [1], and we have very pressing problems now. ¶
但我们迟早会面对这一威胁,而且在它到来之前,我们有大量工作要做。所以这里提出一个建议。 But we will face this threat at some point, and we have a lot of work to do before it gets here. So here is a suggestion. ¶
美国政府以及所有其他政府,都应当对 SMI 的开发进行监管。在一个理想的世界里,监管会拖慢坏人的脚步、加快好人的脚步——最先开发出来的 SMI 会发生什么,看起来将至关重要。 The US government, and all other governments, should regulate the development of SMI. In an ideal world, regulation would slow down the bad guys and speed up the good guys—it seems like what happens with the first SMI to be developed will be very important. ¶
尽管我总体的看法是技术常常被过度监管,但我认为一定的监管是好事,我也不愿意生活在一个完全没有监管的世界里。而当人类的生存受到威胁时,我认为监管绝对是一件好事。(顺带一提,当私人开发的知识对人类生命构成大规模风险时,对其进行保密分级是有先例的。SILEX 也许是最著名的例子。) Although my general belief is that technology is often over-regulated, I think some regulation is a good thing, and I’d hate to live in a world with no regulation at all. And I think it’s definitely a good thing when the survival of humanity is in question. (Incidentally, there is precedent for classification of privately-developed knowledge when it carries mass risk to human life. SILEX is perhaps the best-known example.) ¶
说句显而易见的话:最大的挑战之一是,过去几年里美国政府已经彻底失去了科技社区的信任。要做这件事,我们需要一个新机构。 To state the obvious, one of the biggest challenges is that the US has broken all trust with the tech community over the past couple of years. We’d need a new agency to do this. ¶
我确信网络评论者会说,我即将提出的一切都远远不够具体——这当然没错。我的本意是让它成为一场对话的开始,而不是结束。 I am sure that Internet commentators will say that everything I’m about to propose is not nearly specific enough, which is definitely true. I mean for this to be the beginning of a conversation, not the end of one. ¶
来自 SMI 的最初的严重危险,很可能涉及人类与 SMI 协同行动。监管应当同时应对两种情况:一是恶意之人蓄意滥用机器智能,比如在全球金融市场或空中交通管制系统中制造大破坏;二是「事故」情形——SMI 被开发出来之后行为不可预测。 The first serious dangers from SMI are likely to involve humans and SMI working together. Regulation should address both the case of malevolent humans intentionally misusing machine intelligence to, for example, wreak havoc on worldwide financial markets or air traffic control systems, and the “accident” case of SMI being developed and then acting unpredictably. ¶
具体而言,监管应当: Specifically, regulation should: ¶
1)提供一个观察进展的框架。这应当通过两种方式进行。第一种是留意世界上那些看起来正在获得重大机器智能协助、或以某种方式训练这种智能的团体。 1) Provide a framework to observe progress. This should happen in two ways. The first is looking for places in the world where it seems like a group is either being aided by significant machine intelligence or training such an intelligence in some way. ¶
第二种是观察正在从事 SMI 开发的公司。这些公司不应被迫披露它们做事的具体方法(不过,一旦各国政府对 SMI 认真起来,它们手里的资源很可能超过任何一家私营公司),但定期向监管者展示自己当前的能力,看起来是个明智的做法。 The second is observing companies working on SMI development. The companies shouldn’t have to disclose how they’re doing what they’re doing (though when governments gets serious about SMI they are likely to out-resource any private company), but periodically showing regulators their current capabilities seems like a smart idea. ¶
2)鉴于一个 bug 可能造成的灾难性后果,要求设置开发保障措施,以降低「事故」情形的风险。例如,在超过某个检查点之后,我们可以要求开发只能在与外界物理隔离(airgapped)的计算机上进行;要求自我改进软件在每一轮迭代推进时都必须有人类介入;要求软件的某些部分接受第三方代码审查,等等。除了意外错误之外,我对这些措施能否起任何作用并不太乐观——人类永远会是这套策略中的薄弱环节(参见「AI 装箱」思想实验)。但至少,这感觉值得一试。 2) Given how disastrous a bug could be, require development safeguards to reduce the risk of the accident case. For example, beyond a certain checkpoint, we could require development happen only on airgapped computers, require that self-improving software require human intervention to move forward on each iteration, require that certain parts of the software be subject to third-party code reviews, etc. I’m not very optimistic than any of this will work for anything except accidental errors—humans will always be the weak link in the strategy (see the AI-in-a-box thought experiments). But it at least feels worth trying. ¶
要做到这一点——如果它果真有可能做到——需要大量的技术研发,我们应该现在就开始密集投入。这项工作与今天为让零散的机器智能运转起来所做的工作几乎完全无关。 Being able to do this—if it is possible at all—will require a huge amount of technical research and development that we should start intensive work on now. This work is almost entirely separate from the work that’s happening today to get piecemeal machine intelligence to work. ¶
说一个显而易见但重要的点:在撰写监管规则时,重要的是让它们既提供保护,又对创新造成最小的拖累(尽管会有一些不可避免的成本)。 To state the obvious but important point, it’s important to write the regulations in such a way that they provide protection while producing minimal drag on innovation (though there will be some unavoidable cost). ¶
3)要求最先开发出的 SMI 把以下内容作为其运行规则的一部分:a)它不得对人类造成任何直接或间接的伤害(即阿西莫夫的「第零定律」);b)它应当探测其他正在被开发的 SMI,但除了探测之外不采取任何行动;c)除 b 项所要求之外,对世界不产生任何影响。 3) Require that the first SMI developed have as part of its operating rules that a) it can’t cause any direct or indirect harm to humanity (i.e. Asimov’s zeroeth law), b) it should detect other SMI being developed but take no action beyond detection, c) other than required for part b, have no effect on the world. ¶
我们目前不知道如何实现其中任何一条,所以在这里,我们同样需要重大的技术研发,而且应当现在就启动。 We currently don’t know how to implement any of this, so here too, we need significant technical research and development that we should start now. ¶
4)为遵守全部这些要求的团体提供大量研发资金,尤其要资助从事安全研究的团体。 4) Provide lots of funding for R+D for groups that comply with all of this, especially for groups doing safety research. ¶
5)提供一个更长期的框架,用来思考我们如何为与 SMI 共存找到一个安全而幸福的未来——最乐观的版本似乎是某种形式的「人类/机器融合」。我们不必今天就把这个问题想清楚。 5) Provide a longer-term framework for how we figure out a safe and happy future for coexisting with SMI—the most optimistic version seems like some version of “the human/machine merge”. We don’t have to figure this out today. ¶
监管将通过资金渠道对 SMI 开发产生影响——大多数风险投资公司和大型科技公司都不想违反重大法律。大多数拿了风险投资的创业公司和大公司想必都会遵守这些法规。 Regulation would have an effect on SMI development via financing—most venture firms and large technology companies don’t want to break major laws. Most venture-backed startups and large companies would presumably comply with the regulations. ¶
尽管有可能是车库里的独狼最终搞出 SMI,但更可能的情况是:它将出自一群非常聪明、资源充足的人之手。而且至少就我所了解的当前工作来看,它很可能以某种方式涉及美国公司(不过正如我上面所说,我认为世界上每个政府都应该颁布类似的法规)。 Although it’s possible that a lone wolf in a garage will be the one to figure SMI out, it seems more likely that it will be a group of very smart people with a lot of resources. It also seems likely, at least given the current work I’m aware of, it will involve US companies in some way (though, as I said above, I think every government in the world should enact similar regulations). ¶
有人担心监管会拖慢美国的进展,确保 SMI 在其他地方被率先开发出来。我不认为一点点监管就能够抵消美国公司目前拥有的巨大先发优势和人才密度。 Some people worry that regulation will slow down progress in the US and ensure that SMI gets developed somewhere else first. I don’t think a little bit of regulation is likely to overcome the huge head start and density of talent that US companies currently have. ¶
SMI 有一个显而易见的上行情景——它可以解决人类面临的许多严重问题——但在我看来,这并不是默认情景。另一个大的上行情景是,机器智能可以帮我们弄清如何把自己上传,从而在计算机里永远活下去。又或者,我们可以用某种方式,让 SMI 成为人类的后裔。 There is an obvious upside case to SMI —it could solve a lot of the serious problems facing humanity—but in my opinion it is not the default case. The other big upside case is that machine intelligence could help us figure out how to upload ourselves, and we could live forever in computers. Or maybe in some way, we can make SMI be a descendent of humanity. ¶
总的来说,技术发展的弧线一直是减少随机性、增强我们对世界的控制。而在未来一个世纪里的某个时点,我们将把有史以来最大的随机性注入这个系统。 Generally, the arc of technology has been about reducing randomness and increasing our control over the world. At some point in the next century, we are going to have the most randomness ever injected into the system. ¶
在政治中,我们通常为了微小的分歧而争斗。与人类和外星人之间的差异相比,这些分歧黯然失色——而 SMI 实际上就会像是外星人。我们应该能够团结起来,迅速制定出一套监管战略。 In politics, we usually fight over small differences. These differences pale in comparison to the difference between humans and aliens, which is what SMI will effectively be like. We should be able to come together and figure out a regulatory strategy quickly. ¶
感谢 Dario Amodei(尤其是 Dario)、Paul Buchheit、Matt Bush、Patrick Collison、Holden Karnofsky、Luke Muehlhauser 和 Geoff Ralston 阅读本文和上一篇的草稿。 Thanks to Dario Amodei (especially Dario), Paul Buchheit, Matt Bush, Patrick Collison, Holden Karnofsky, Luke Muehlhauser, and Geoff Ralston for reading drafts of this and the previous post. ¶
[1] 如果你想试着猜测时间,我会考虑的两个因素是计算能力和算法进展。就前者而言,假设人脑约有 1000 亿个神经元和 100 万亿个突触,平均每个神经元每秒放电 5 次,然后想一想:按当前的计算发展轨迹,要多久才能得到一台拥有足够内存和浮点运算能力(flops)来模拟这一切的机器。 [1] If you want to try to guess when, the two things I’d think about are computational power and algorithmic development. For the former, assume there are about 100 billion neurons and 100 trillion synapses in a human brain, and the average neuron fires 5 times per second, and then think about how long it will take on the current computing trajectory to get a machine with enough memory and flops to simulate that. ¶
就算法而言,神经网络和强化学习分别在输入和输出上的表现都超出了我的预期(例如:给描绘复杂场景的照片配写说明文字;在软件从未见过的电子游戏里击败人类——它只有看屏幕的能力和操作控制器的权限)。我总是不明白,为什么大多数人对这些结果显得不以为然。无监督学习一直是较薄弱的一环,而这很可能是复现人类智能的关键部分。但与我交谈过的许多研究者都对当前的工作持乐观态度,而且我没有理由相信这超出了图灵机的能力范围。 For the algorithms, neural networks and reinforcement learning have both performed better than I’ve expected for input and output respectively (e.g. captioning photos depicting complex scenes, beating humans at video games the software has never seen before with just the ability to look at the screen and access to the controls). I am always surprised how unimpressed most people seem with these results. Unsupervised learning has been a weaker point, and this is probably a critical part of replicating human intelligence. But many researchers I’ve spoken to are optimistic about current work, and I have no reason to believe this is outside the scope of a Turing machine. ¶