From: Mitch Howe (mitch@iconfound.com)
Date: Sun May 30 2004 - 09:53:37 MDT
Philip Sutton and Mark Waser wrote:
-- (that multiple AGI is better than one) -- You both believe that checks and balances are important. You both seem to understand that an AGI will be immensely powerful. And you are both correct. However, by suggesting that creating multiple, different new minds would make us safer you are fighting the correct battle in the wrong country. *Any* successfully launched AGI launched in isolation will have the keys to the Solar System. Being vastly more intelligent than any other entity around does that to you. But you speak as if *different* AGIs could be sent ahead together, like three explorers in a dingy rowing out from the mother ship to a new shore -- all with something to gain from cooperation, and all free to throttle the others at will if either threatens to horde El Dorado for himself. Nope. Different AGIs would launch into new domains of intelligence at different speeds and along different trajectories. At any given moment, one would almost certainly be far more powerful than any other. Instead of three explorers in a dingy, you would have Tiny Tim in an inner-tube, Paul Bunyan on a hovercraft, and a Borg Hive on a Death Star. No heated discussions. No family council. Not even a clash of the titans. No contest at all. So yes, there is a very important time for checks and balances, and that time is before any AGI program goes live. Yes, programmers are flawed. Success must be ensured in spite of this. This is what Friendliness theory is all about, and one of the reasons it is so damned tricky. And, not coincidentally, it is the reason why anyone taking it seriously right now doesn't have any code to show for it. Solution first, implementation later. --Mitch Howe
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT