evil and EVIL

fabrisse has recently been musing in her LJ about the nature of evil. At one point in the discussion, she quoted a bit of Terry Pratchett that she said she agreed with:

The trouble is, you see, that if you do know Right from Wrong you can't choose Wrong. You just can't do it and live. Maskerade

I think this is only partially right, and perhaps dangerously misleading. The weakness is hinted at by the capital letter in "Wrong". Yes, if you know something is Wrong enough that you spell it with a capital letter, then I agree that you probably can't do it. But people do small-w wrong things all the time (such as posting on LJ when they're theoretically supposed to be working). And it's a slippery slope.

For example, exaggeration is generally perceived as a small-w wrong. You can exaggerate a bit and still sleep at night. Especially if the exaggeration is in support of a basic principle that you believe to be accurate. But just such exaggeration is what got us into Iraq, where hundreds of soldiers, and tens of thousands of Iraqi innocents, have died unnecessarily. Most readers of this journal probably would agree that that is a big-W Wrong.

Many years ago, I learned of an early interesting result in the field of Information Theory. Given a system that can adjust its priorities dynamically *at all*, even if the only adjustments allowed are EXTREMELY SMALL, it is easy to show mathematically that the system is capable of totally upending and reversing its initial prioirty ordering. In just such a way, by incremental steps of wrong and wrong and wrong, do we arrive at length at WRONG, while never realizing how far we've gone. Not all of us, of course, but many.

Once someone has taken all those small-w steps that arrive at last at (what they once would have seen as) WRONG, they are very unlikely to be able to perceive those capital letters any more. Realization and repentance remain possible, but they're not the way to bet. Meanwhile, they sleep like babies, secure in their righteousness.

This is true on an institutional scale as well as an individual one. The Catholic Church didn't go from "Thou Shalt Not Kill", to the concept of Crusade, to "Kill them all; God will know his own" in a single step, but by many small increments.

[Incidentally, this is part of why I don't believe that Asimov's Laws of Robotics are actually implementable. Any system that was that close to human intelligence would necessarily have dynamic prioritization, and would thus be susceptible to runaway reconfiguration. You could use human-style indoctrination techniques, but they would only have human-scale success rates. Most people raised Christian follow most of the ten commandments most of the time. But you still get the occasional serial killer. Or worse, the occasional Prophet.]

evil and EVIL

fabrisse has recently been musing in her LJ about the nature of evil. At one point in the discussion, she quoted a bit of Terry Pratchett that she said she agreed with:

The trouble is, you see, that if you do know Right from Wrong you can't choose Wrong. You just can't do it and live. Maskerade

I think this is only partially right, and perhaps dangerously misleading. The weakness is hinted at by the capital letter in "Wrong". Yes, if you know something is Wrong enough that you spell it with a capital letter, then I agree that you probably can't do it. But people do small-w wrong things all the time (such as posting on LJ when they're theoretically supposed to be working). And it's a slippery slope.

For example, exaggeration is generally perceived as a small-w wrong. You can exaggerate a bit and still sleep at night. Especially if the exaggeration is in support of a basic principle that you believe to be accurate. But just such exaggeration is what got us into Iraq, where hundreds of soldiers, and tens of thousands of Iraqi innocents, have died unnecessarily. Most readers of this journal probably would agree that that is a big-W Wrong.

Many years ago, I learned of an early interesting result in the field of Information Theory. Given a system that can adjust its priorities dynamically *at all*, even if the only adjustments allowed are EXTREMELY SMALL, it is easy to show mathematically that the system is capable of totally upending and reversing its initial prioirty ordering. In just such a way, by incremental steps of wrong and wrong and wrong, do we arrive at length at WRONG, while never realizing how far we've gone. Not all of us, of course, but many.

Once someone has taken all those small-w steps that arrive at last at (what they once would have seen as) WRONG, they are very unlikely to be able to perceive those capital letters any more. Realization and repentance remain possible, but they're not the way to bet. Meanwhile, they sleep like babies, secure in their righteousness.

This is true on an institutional scale as well as an individual one. The Catholic Church didn't go from "Thou Shalt Not Kill", to the concept of Crusade, to "Kill them all; God will know his own" in a single step, but by many small increments.

[Incidentally, this is part of why I don't believe that Asimov's Laws of Robotics are actually implementable. Any system that was that close to human intelligence would necessarily have dynamic prioritization, and would thus be susceptible to runaway reconfiguration. You could use human-style indoctrination techniques, but they would only have human-scale success rates. Most people raised Christian follow most of the ten commandments most of the time. But you still get the occasional serial killer. Or worse, the occasional Prophet.]