From: Thomas Buckner (tcbevolver@yahoo.com)
Date: Mon Oct 26 2009 - 22:17:08 MDT
----- Original Message ----
From: Matt Mahoney <matmahoney@yahoo.com>
To: sl4@sl4.org
Sent: Mon, October 26, 2009 9:24:28 AM
Subject: [sl4] Is friendliness computable? (was Re: Why extrapolate?)
Tim Freeman brought this paper by Peter de Blanc to our attention http://arxiv.org/abs/0907.5598
It my understanding is correct, it dooms any hope that a computable mind could experience unbounded happiness, where "happiness" is defined as an increase in utility. I argued previously that in any finite state machine whose behavior can be modeled in terms of a utility function, that there is a state of maximum utility in which any thought or perception would be unpleasant because it would result in a different state. It should be obvious that this degenerate state of Utopian bliss that all goal seeking agents aspire to (whether they know it or not) is indistinguishable from death.
A way out would be to continually add memory to your mind so that the number of states is unbounded. De Blanc's paper quashes that approach. Utility must be bounded or else you cannot make choices based on expected utility. De Blanc discusses some ways out in section 6, but these all involve alternative utility functions, all of which are bounded.
Perhaps, then, friendliness is not computable, and we should just give up and let evolution do what it's going to do.
-- Matt Mahoney, matmahoney@yahoo.com
I had the impression that what you say was proven years ago, except that adding memory never really equated to being "happier" anyway, but rather equates to opening more options for possible mental states. Adding more possible states needn't move the settings for either max or minimum values for any condition, but does add gradations between the extremes. In any case, Happiness, however defined, doesn't equate to Friendliness in any obvious way. I do agree that stasis at any maximum of bliss is isomorphic to death. Everything moves. This is old news to careful readers of zen and taoist thought, which is bizarrely far ahead of its time.
Tom Buckner
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:05 MDT