Re: Ethical experimentation on AIs

From: Matt Mahoney (
Date: Sun Oct 21 2007 - 11:01:57 MDT

--- Chris Hibbert <> wrote:

> Evolution has little to do with what we "ought" to do.

Yes it does. Ethics is a function of your upbringing. Cultures that teach
respect for other humans in your cultural group are more successful than those
that don't.

> My theory has long been that self-awareness is the level that I would
> demand from any creature before granting it an equal "right to live".
> Others seem to rely on ability to feel pain, or something even less.

How do you test for self awareness?

> > Also, do you presume there is a test to distinguish a machine that can
> feel
> > pain from one that only claims to feel pain?
> If the claim seems real, and not recorded, I'm willing to count that.

What makes a claim seem real? If you can trace the neural pathways in my
brain when I say "ouch" and conclude that my response was computable, is the
pain still real? If a program says "ouch" but the code is too complex for you
to understand, is its pain real?

If a program changes its behavior to avoid a negative reinforcement signal, as
the program below does, does it experience pain? If not, then what test does
this program fail?

/* This program simulates a programmable 2 input logic gate
that is trained by reinforcement learning. It can be trained
to implement any 2 input function (AND, OR, XOR, NAND, etc). */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
int main() {
  static double mem[4]={0,0,0,0}; // tendency to output 1
  while (1) {
    int input; // 0..3 meaning 00, 01, 10, 11
    int output; // 0 or 1
    double reward; // reinforcement signal
    printf("Enter input (00, 01, 10, or 11)? ");
    scanf("%d", &input);
    input-=8*(input>=10); // 00, 01, 10, 11 --> 0, 1, 2, 3
    if (input>=0 && input<=3) {
      output=RAND_MAX > rand()*(1+exp(-mem[input]));
      printf("Output = %d. ", output);
      printf("Enter reward (positive) or penalty (negative) ");
      scanf("%lf", &reward);
      mem[input]+=reward*(2*output-1); // learn!
      if (reward>3) printf("Oh yes!!!\n"); // act human :-)
      else if (reward>0.5) printf("aah!\n");
      else if (reward>-0.5) printf("OK\n");
      else if (reward>-1.5) printf("Oh!\n");
      else if (reward>-3) printf("Ouch!\n");
      else if (reward>-10) printf("AAARGH!!!\n");
      else printf("Fatal error\n"), exit(1);

-- Matt Mahoney,

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT