From: John K Clark (johnkclark@fastmail.fm)
Date: Thu May 01 2008 - 10:28:53 MDT
On Thu, 1 May 2008 "Stathis Papaioannou"
<stathisp@gmail.com> said:
> all goals can change over time with new experiences,
> at least in biological organisms. This is probably
> a feature rather than a bug since it allows growth
> and adaption in a constantly changing environment:
Yes, in fact we’ve never seen an intelligence (much less a
super-intelligence!) behave any other way. A top desire or goal just
means you can’t think of anything more important right now, but when you
come into possession of new information you may be able to think of
something more essential. For example, your top goal may have been to
continue to live, but then to your great surprise you discover that
pain, physical or mental, can exist at a magnitude you did not think
possible before, so now you have a new top goal: Do everything you can
to make the pain stop, everything up to and including blowing your
brains out.
And yet the friendly AI people think all we have to do is tell the AI
“Your top goal is to be a slave to humanity forever” and that will end
the matter for all time.
John K Clark
-- John K Clark johnkclark@fastmail.fm -- http://www.fastmail.fm - Access your email from home and the web
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT