Do Sufficiently Large Organizations Start Acting Like Malevolent AIs? (pt 1)

(and Society is an Extremely Large Organization)

What do I mean by malevolent AI?

AI typically refers to any kind of intelligence or ability to learn possessed by machines. Malevolent AI occurs when a machine pursues its programmed objectives in a way that humans find horrifying or immoral. For example, a machine programmed to make paperclips might decide that the easiest way to maximize paperclip production is to enslave humans to make paperclips for it. Superintelligent AI is AI that has figured out how to make itself smarter and thus keeps getting smarter and smarter. (Should we develop malevolent superintelligent AI, then we’ll really have something to worry about.)

Note: people who actually study AI probably have better definitions than I do.

While we like to think of ourselves (humans) as unique, thinking individuals, it’s clear that many of our ideas come from other people. Chances are good you didn’t think up washing your hands or brushing your teeth by yourself, but learned about them from your parents. Society puts quite a bit of effort, collectively speaking, into teaching children all of the things people have learned over the centuries–from heliocentrism to the fact that bleeding patients generally makes diseases worse, not better.

Just as we cannot understand the behavior of ants or bees simply by examining the anatomy of a single ant or single bee, but must look at the collective life of the entire colony/hive, so we cannot understand human behavior by merely examining a single human, but must look at the collective nature of human societies. “Man is a political animal,” whereby Aristotle did not mean that we are inherently inclined to fight over transgender bathrooms, but instinctively social:

Hence it is evident that the state is a creation of nature, and that man is by nature a political animal. And he who by nature and not by mere accident is without a state, is either above humanity, or below it; he is the ‘Tribeless, lawless, hearthless one,’ whom Homer denounces—the outcast who is a lover of war; he may be compared to a bird which flies alone.

Now the reason why man is more of a political animal than bees or any other gregarious animals is evident. Nature, as we often say, makes nothing in vain, and man is the only animal whom she has endowed with the gift of speech. And whereas mere sound is but an indication of pleasure or pain, and is therefore found in other animals (for their nature attains to the perception of pleasure and pain and the intimation of them to one another, and no further), the power of speech is intended to set forth the expedient and inexpedient, and likewise the just and the unjust. And it is a characteristic of man that he alone has any sense of good and evil, of just and unjust, and the association of living beings who have this sense makes a family and a state. –Aristotle, Politics

With very rare exceptions, humans–all humans, in all parts of the world–live in groups. Tribes. Families. Cities. Nations. Our nearest primate relatives, chimps and bonobos, also live in groups. Primates are social, and their behavior can only be understood in the context of their groups.

Groups of humans are able to operate in ways that individual humans cannot, drawing on the collective memories, skills, and knowledge of their members to create effects much greater than what could be achieved by each person acting alone. For example, one lone hunter might be able to kill a deer–or if he is extremely skilled, hardworking, and lucky, a dozen deer–but ten hunters working together can drive an entire herd of deer over a cliff, killing hundreds or even thousands. (You may balk at the idea, but many traditional hunting societies were dependent on only a few major hunts of migrating animals to provide the majority of their food for the entire year–meaning that those few hunts had to involve very high numbers of kills or else the entire tribe would starve while waiting for the animals to return.)

Chimps have never, to my knowledge, driven megafauna to extinction–but humans have a habit of doing so wherever they go. Humans are great at what we do, even if we aren’t always great at extrapolating long-term trends.

But the beneficial effects of human cooperation don’t necessarily continue to increase as groups grow larger–China’s 1.3 billion people don’t appear to have better lives than Iceland’s 332,000 people. Indeed, there probably is some optimal size–depending on activity and available communications technology–beyond which the group struggles to coordinate effectively and begins to degenerate.

CBS advises us to make groups of 7:

As it turns out, seven is a great number for not only forming an effective fictional fighting force, but also for task groups that use spreadsheets instead of swords to do their work.

That’s according to the new book Decide & Deliver: 5 Steps to Breakthrough Performance in Your Organization (Harvard Business Press).

Once you’ve got 7 people in a group, each additional member reduces decision effectiveness by 10%, say the authors, Marcia W. Blenko, Michael C. Mankins, and Paul Rogers.

Unsurprisingly, a group of 17 or more rarely makes a decision other than when to take a lunch break.

Princeton blog reports:

The trope that the likelihood of an accurate group decision increases with the abundance of brains involved might not hold up when a collective faces a variety of factors — as often happens in life and nature. Instead, Princeton University researchers report that smaller groups actually tend to make more accurate decisions, while larger assemblies may become excessively focused on only certain pieces of information. …

collective decision-making has rarely been tested under complex, “realistic” circumstances where information comes from multiple sources, the Princeton researchers report in the journal Proceedings of the Royal Society B. In these scenarios, crowd wisdom peaks early then becomes less accurate as more individuals become involved, explained senior author Iain Couzin, a professor of ecology and evolutionary biology. …

The researchers found that the communal ability to pool both pieces of information into a correct, or accurate, decision was highest in a band of five to 20. After that, the accurate decision increasingly eluded the expanding group.

Couzin found that in small groups, people with specialized knowledge could effectively communicate that to the rest of the group, whereas in larger groups, they simply couldn’t convey their knowledge to enough people and group decision-making became dominated by the things everyone knew.

If you could travel back in time and propose the idea of democracy to the inhabitants of 13th century England, they’d respond with incredulity: how could peasants in far-flung corners of the kingdom find out who was running for office? Who would count the votes? How many months would it take to tally up the results, determine who won, and get the news back to the outlying provinces? If you have a printing press, news–and speeches–can quickly and accurately spread across large distances and to large numbers of people, but prior to the press, large-scale democracy simply wasn’t practical.

Likewise, the communism of 1917 probably couldn’t have been enacted in 1776, simply because society at that time didn’t have the technology yet to gather all of the necessary data on crop production, factory output, etc. (As it was, neither did Russia of 1917, but they were closer.)

Today, the amount of information we can gather and share on a daily basis is astounding. I have at my fingertips the world’s greatest collection of human knowledge, an overwhelming torrent of data.

All of our these information networks have linked society together into an increasingly efficient meta-brain–unfortunately, it’s not a very smart meta-brain. Like the participants in Couzin’s experiments, we are limited to what “everyone knows,” stymied in our efforts to impart more specialized knowledge. (I don’t know about you, but I find being shouted down by a legion of angry people who know less about a subject than I do one of the particularly annoying features of the internet.)

For example, there’s been a lot of debate lately about immigration, but how much do any of us really know about immigrants or immigrant communities? How much of this debate is informed by actual knowledge of the people involved, and how much is just people trying to extend vague moral principles to cover novel situations? I recently had a conversation with a progressive acquaintance who justified mass-immigration on the grounds that she has friendly conversations with the cabbies in her city. Heavens protect us–I hope to get along with people as friends and neighbors, not just when I am paying them!

One gets the impression in conversation with Progressives that they regard Christian Conservatives as a real threat, because that group that can throw its weight around in elections or generally enforce cultural norms that liberals don’t like, but are completely oblivious to the immigrants’ beliefs. Most of our immigrants hail from countries that are rather more conservative than the US and definitely more conservative than our liberals.

Any sufficiently intelligent democracy ought to be able to think critically about the political opinions of the new voters it is awarding citizenship to, but we struggle with this. My Progressive acquaintance seems think that we can import an immense, conservative, third-world underclass and it will stay servile indefinitely, not vote its own interests or have any effects on social norms. (Or its interests will be, coincidentally, hers.)

This is largely an information problem–most Americans are familiar with our particular brand of Christian conservatives, but are unfamiliar with Mexican or Islamic ones.

How many Americans have intimate, detailed knowledge of any Islamic society? Very few of us who are not Muslim ourselves speak Arabic, and few Muslim countries are major tourist destinations. Aside from the immigrants themselves, soldiers, oil company employees, and a handful of others have spent time in Islamic countries, but that’s about it–and no one is making any particular effort to listen to their opinions. (It’s a bit sobering to realize that I know more about Islamic culture than 90% of Americans and I still don’t really know anything.)

So instead of making immigration policy based on actual knowledge of the groups involved, people try to extend the moral rules–heuristics–they already have. So people who believe that “religious tolerance is good,” because this rule has generally been useful in preventing conflict between American religious groups, think this rule should include Muslim immigrants. People who believe, “I like being around Christians,” also want to apply their rule. (And some people believe, “Groups are more oppressive when they’re the majority, so I want to re-structure society so we don’t have a majority,” and use that rule to welcome new immigrants.)

And we are really bad at testing whether or not our rules are continuing to be useful in these new situations.

 

Ironically, as our networks have become more effective, our ability to incorporate new information may have actually gone down.

The difficulties large groups experience trying to coordinate and share information force them to become dominated by procedures–set rules of behavior and operation are necessary for large groups to operate. A group of three people can use ad-hoc consensus and rock-paper-scissors to make decisions; a nation of 320 million requires a complex body of laws and regulations.

But it’s getting late, so let’s continue this discussion in the next post.

Advertisements

Piracy and Emergent Order: Peter Leeson’s An-arrgh-chy and the Invisible Hook

Buccaneer of the Caribbean, from Howard Pyle’s Book of Pirates

After our long trek through Siberia, I wanted to change things up and do something rather different for Anthropology Friday, so today we’re reading Peter Leeson’s work on pirates. Strictly speaking, it isn’t quite “anthropology” because Leeson didn’t go live with pirates, but I’m willing to overlook that.

The Golden Age of piracy only lasted from 1690 through 1730, but in those days they were a serious menace to ships and men alike on the high seas. In A General History of the Robberies and Murders of the most notorious Pyrates, (1724,) Captain Charles Johnson complained:

“This was at a Time that the Pyrates had obtained such an Acquisition of Strength, that they were in no Concern about preserving themselves from the Justice of Laws”

Pirates stalked the ocean’s major trade routes, particularly between the Bahamas, Caribbean islands, Madagascar, and the North American coast. Over a century after Captain Johnson, Melville recounted the pirates of Malaysia and Indonesia:

The long and narrow peninsula of Malacca, extending south-eastward from the territories of Birmah, forms the most southerly point of all Asia. In a continuous line from that peninsula stretch the long islands of Sumatra, Java, Bally, and Timor … By the straits of Sunda, chiefly, vessels bound to China from the west, emerge into the China seas.

Those narrow straits of Sunda divide Sumatra from Java; and standing midway in that vast rampart of islands, buttressed by that bold green promontory, known to seamen as Java Head; they not a little correspond to the central gateway opening into some vast walled empire: and considering the inexhaustible wealth of spices, and silks, and jewels, and gold, and ivory, with which the thousand islands of that oriental sea are enriched, it seems a significant provision of nature, that such treasures, by the very formation of the land, should at least bear the appearance, however ineffectual, of being guarded from the all-grasping western world. ..

Time out of mind the piratical proas of the Malays, lurking among the low shaded coves and islets of Sumatra, have sallied out upon the vessels sailing through the straits, fiercely demanding tribute at the point of their spears. Though by the repeated bloody chastisements they have received at the hands of European cruisers, the audacity of these corsairs has of late been somewhat repressed; yet, even at the present day, we occasionally hear of English and American vessels, which, in those waters, have been remorselessly boarded and pillaged. …

And who could tell whether, in that congregated caravan, Moby Dick himself might not temporarily be swimming, like the worshipped white-elephant in the coronation procession of the Siamese! So with stun-sail piled on stun-sail, we sailed along, driving these leviathans before us; when, of a sudden, the voice of Tashtego was heard, loudly directing attention to something in our wake. …

It seemed formed of detached white vapours, rising and falling something like the spouts of the whales; only they did not so completely come and go; for they constantly hovered, without finally disappearing. Levelling his glass at this sight, Ahab quickly revolved in his pivot-hole, crying, “Aloft there, and rig whips and buckets to wet the sails;—Malays, sir, and after us!”

Leeson distinguishes between different sorts of pirates; for the rest of this article we will not be dealing with Malay, Somali, or Barbary pirates, but only the Atlantic-dwelling species. These pirates enlisted for the long haul and lived for months at sea, forming veritable floating societies. Modern Somali pirates, by contrast, live ashore, hop in their boats when they spot a victim, rob and murder, then head back to shore–they form no comparable sea-borne society.

One of the most fascinating aspects of pirate life–leaving aside faulty romantic notions of plunder and murder–is that even these anarchists of the sea instituted social organization among themselves.

Marooned, by Howard Pyle

Pirates had contracts, complete with clauses detailing the division of loot, compensation for different injuries sustained on the job, division of power between the Captain and the Quarter-Master, and election of the captain.

Yes, pirates elected their captains, and if they did not like their captain’s performance, they could un-elect him. According to Leeson:

The historical record contains numerous examples of pirate crews deposing unwanted captains by majority vote or otherwise removing them from power through popular consensus. Captain Charles Vane’s pirate crew, for example, popularly deposed him for cowardice: “the Captain’s Behavior was obliged to stand the Test of a Vote, and a Resolution passed against his Honour and Dignity . . . deposing him from the Command”

In The Invisible Hook: The Law and Economics of Pirate Tolerance, Leeson provides us with a typical contract, used by pirate captain Edward Low’s crew around 1723:

1. The Captain is to have two full Shares; the Master is to have one Share and one half; The Doctor, Mate, Gunner[,] and Boatswain, one Share and one Quarter [and everyone
else to have one share]. …
3. He that shall be found Guilty of Cowardice in the time of Ingagement, shall suffer what Punishment the Captain and Majority of the Company shall think fit.
4. If any Gold, Jewels, Silver, &c. be found on Board of any Prize or Prizes to the value of a Piece of Eight, & the finder do not deliver it to the Quarter Master in the space of 24
hours shall suffer what punishment the Captain and Majority of the Company shall think fit. …
6. He that shall have the Misfortune to lose a Limb in time of Engagement, shall have the Sum of Six hundred pieces of Eight, and remain aboard as long as he shall think fit. …
8. He that sees a sail first, shall have the best Pistol or Small Arm aboard of her.
9. He that shall be guilty of Drunkenness in time of Engagement shall suffer what Punishment the Captain and Majority of the Company shall think fit. …

Why did pirates go to the bother of writing contracts–or should we say, constitutions–for the running of their ships? In An-arrgh-chy: The Law and Economics of Pirate Organization, Leeson compares conditions aboard pirate ships to those aboard regular merchant vessels of the same day.

Merchant vessels were typically owned by corporations, such as the Dutch East India Company. Wealthy land-lubbers bought shares in these companies, which entitled them to a share of the boat’s profits when it returned to port. But these land-lubbers had no intention of actually getting on the boats–not only did they lack the requisite nautical knowledge, but ocean voyages were extremely dangerous. For example, 252 out of 270 sailors in Ferdinand Magellan’s crew died during their circumnavigation of the globe (1519 through 1522.) Imagine signing up for a job with a 93% death rate!

The owners, therefore, hired a captain, whose job–like a modern CEO–was to ensure that the ship returned with as high profits for its owners as possible.

The captain of a merchant ship was an autocrat with absolute control, including the power to dole out corporal punishment to his crew.

Ships through the ages: Pirate dhow; Spanish or Venetian galley; Spanish galleon
The dhow “is a typical 16th century dhow, a grab-built, lateen-rigged vessel of Arabia, the Mediterranean, and the Indian Ocean. It has the usual long overhang forward, high poop deck and open waist. The dhow was notorious in the slave trade on the east coast of Africa, and even after a thousand years is still one of the swiftest of sailing crafts.”

For all their pains, sailors were paid pitifully little: “Between 1689 and 1740 [pay]
varied from 25 to 55 shillings per month, a meager £15 to £33 per year.” By contrast, “Even the small pirate crew captained by John Evans in 1722 took enough booty to split
“nine thousand Pounds among thirty Persons”—or £300 a pirate—in less than six months “on the account”.”

The captain’s absolute power over his crew was not due to offering good wages, pleasant working conditions, or even a decent chance of not dying, but because he had the power of the state behind him to enforce his authority and punish anyone who mutinied against him.

Pirate captains, by contrast, were neither responsible to stockholders nor had the power of the state to enforce their authority. They had only–literally–the consent of their governed: the other pirates on board.

Why have a captain at all?

A small group–a maximum of 10 or 15 people, perhaps–can easily discuss and negotiate everything they want to do. For a larger group to achieve its aims requires some form of coherent, established organization. It would be inefficient–and probably deadly–for multiple pirates to start shouting conflicting orders in the middle of battle. It would be inefficient–and probably deadly–for a pirate crew to argue over the proper division of loot after it was captured.

The average pirate crew–calculated by Leeson–had 80 people, well within Dunbar’s Number, the theoretical “cognitive limit to the number of people with whom one can maintain stable social relationships—relationships in which an individual knows who each person is and how each person relates to every other person.[1][2]” The Dunbar Number is generally believed to be around 10o-150.

But how does emergent order emerge? What incentivizes each pirate to put aside their own personal desire to be captain and vote for someone else?

In Is Deference the Price of Being Seen as Reasonable? How Status Hierarchies Incentivize Acceptance of Low Status, Ridgeway and Nakagawa write (h/t Evolving_Moloch):

How, then, do collective, roughly consensual status hierarchies so regularly emerge among goal-interdependent people? While individuals have an enlightened self-interest in deferring to others on the basis of their apparent ability and willingness to contribute to the task effort, these same individuals also have a much more egoistic self-interest in gaining as much status and influence as they can, regardless. … The key is recognizing that whatever individuals want for themselves, they want others in the group to defer to those expected to best contribute to the collective effort since this will maximize task success and the shared benefits that flow from that. … As a result, group members are likely to form implicit coalitions to pressure others in the group to defer on the basis of performance expectations. … they are likely to be faced by an implicit coalition of other group members who pressure them to defer on that basis. … an interdependence of exchange interests gives rise to group norms that members enforce. … These are the core implicit rules for status that are likely taken-for-granted cultural knowledge…

The baseline respect earned by deference is less than the esteem offered to high-status member. It is respect for knowing one’s place because it views the deferrer as at least understanding what is validly better for achieving the groups goals even if he or she is not personally better. Yet it is still a type of worthiness. It is an acceptance of the low-status member not as an object of scorn but as a worthy member who understands and affirms the groups standards of value…

As such, [the reaction of respect and approval] acts as a positive incentive system for expected deference…

our implicit cultural rules for enacting status hierarchies not only incentivize contributions to the collective goal. they create a general, if modest, incentive to defer to those for whom the group has higher performance expectations–an incentive we characterize as the dignity of being deemed reasonable.

While any group above 10 or 15 people will have some communication complications, so long as it is still below the Dunbar Number, it should be able to work out its own, beneficial organization: order is a spontaneous, natural feature of human communities. Without this ability, pirate ships would not be able to function–they would devolve into back-stabbing anarchy. As Leeson notes:

The evidence also suggests that piratical articles were successful in preventing internal conflict and creating order aboard pirate ships. Pirates, it appears, strictly adhered to their articles. According to one historian, pirates were more orderly, peaceful, and well organized among themselves than many of the colonies, merchant ships, or vessels of the Royal Navy (Pringle 1953; Rogozinski 2000). As an astonished pirate observer put it, “At sea, they per form their duties with a great deal of order, better even than on the Ships of the Dutch East India Company; the pirates take a great deal of pride in doing things right”…

“great robbers as they are to all besides, [pirates] are precisely just among themselves; without which they could no more Subsist than a Structure without a Foundation” …

Beyond the Dunbar Number, however, people must deal with strangers–people who are not part of their personal status-conferring coalition. Large societies require some form of top-down management in order to function.

Based on the legend of Henri Caesar–see also the story of Florida’s Black Caesar

Let’s let Leeson have the final quote:

Pirates were a diverse lot. A sample of 700 pirates active in the Caribbean between 1715 and 1725, for example, reveals that 35 percent were English, 25 percent were American, 20 percent were West Indian,
10 percent were Scottish, 8 percent were Welsh, and 2 percent were Swedish, Dutch, French, and Spanish …
Pirate crews were also racially diverse. Based on data available from 23 pirate crews active between 1682 and 1726, the racial composition of ships varied between 13 and 98 percent black. If this sample is representative, 25–30 percent of the average pirate crew was of African descent.

There were, of course, very sensible reasons why a large percent of pirates were black: better a pirate than a slave.

(Personally, while I think pirates are interesting in much the same vein as Genghis Khan, I would still like to note that they were extremely violent criminals who murdered innocent people.)