Re: Normative Reasoning: A Siren Song?

From: Peter C. McCluskey (
Date: Tue Sep 28 2004 - 14:09:45 MDT (Michael Wilson) writes:
>and computing power. If there's any sort of competition going on, turning
>yourself into a RPOP is the way to win. Unfortunately it also appears to be
>the end of most of the stuff we place moral value on. A universe full of
>perfect rationalists is a universe where all diversity resides solely in
>people's goal systems (which may or may not converge); the qualities of
>'insight', 'creativeness', 'willpower' etc will all dissappear as they are
>defined against flaws, and goal-system properties such as 'compassion' will
>revert to 'did this person have an initial utility function that was
>compassionate under renomralisation'? This is on top of the already known

 I'm having trouble figuring out why we should worry about this kind of
problem. We have goal-systems which are sometimes inconsistent, and when
we need to choose between conflicting goals, we figure out which goal is
least important and discard it as an obsolete sub-goal. The qualities you
mention above seem like sub-goals that I would be willing to discard if
they conflicted with more important goals such as happiness.

>The CV question could be glibly summarised as 'is there a likely incremental
>self-improvement path from me to a paperclip optimiser?'. While few people
>like paperclips /that/ much, it seems likely that many people would choose
>to become perfect rationalists without appreciating what they're losing. If

 If better information would cause people to choose a different path, then
that suggests that better information would be desirable. But that's a
normal problem with most decisions, and I don't see how the CV creates
a different type of problem in that regard.
 Or were you suggesting that we should be upset with the end result if
fully informed people would decide to follow that path?

Peter McCluskey          | Asimov's 3 laws of robotics considered harmful; | see

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:49 MDT