From: Matt Mahoney (matmahoney@yahoo.com)
Date: Thu Oct 23 2008 - 20:53:56 MDT
--- On Thu, 10/23/08, Mike Dougherty <msd001@gmail.com> wrote:
> On Thu, Oct 23, 2008 at 7:52 PM, Matt Mahoney
> <matmahoney@yahoo.com> wrote:
> > --- On Thu, 10/23/08, Toby Weston
> <lordlobster@yahoo.com> wrote:
> >> Just in case we do, deep down, want to kill all
> humans.
> >> Perhaps we should add a hardcoded caveat to the
> friendliness
> >> function, that puts all basline, pre-posthuman,
> homo sapiens
> >> off limits to the AGI god's meddling. Let the
> Amish live
> >> whatever happens.
> >
> > Wouldn't it be easier (or at least, have a higher
> probability of getting the expected result) if we just ban
> AI?
>
> To clarify - is the "expected result" to kill all
> humans or not? I
> thought we wanted AI to be smart enough to protect us from
> other eventual AI as well as the myriad non-AI ways humanity
> could wipe itself out.
Expected, as in getting what we think we asked for (which might not be what we asked for). The problem with hardcoding "don't kill all humans" is that these terms aren't well defined in a future world. Is an upload a human? If so, is it "killing" if you kill one copy but not the other? How close does a copy have to be to count as an upload?
A description of friendliness is algorithmically complex. We have no choice but to implement it without defining it, because all definitions that we can understand are wrong.
-- Matt Mahoney, matmahoney@yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT