Do Sufficiently Large Organizations Start Acting Like Malevolent AIs? (pt 2)

(Part 1 is here)

As we were discussing on Monday, as our networks have become more effective, our ability to incorporate new information may have actually gone down. Ironically, as we add more people to a group–beyond a certain limit–it becomes more difficult for individuals with particular expertise to convince everyone else in the group that the group’s majority consensus is wrong.

The difficulties large groups experience trying to coordinate and share information force them to become dominated by procedures–set rules of behavior and operation are necessary for large groups to operate. A group of three people can use ad-hoc consensus and rock-paper-scissors to make decisions; a nation of 320 million requires a complex body of laws and regulations. (I once tried to figure out just how many laws and regulations America has. The answer I found was that no one knows.)

An organization is initially founded to accomplish some purpose that benefits its founders–generally to make them well-off, but often also to produce some useful good or service. A small organization is lean, efficient, and generally exemplifies the ideals put forth in Adam Smith’s invisible hand:

It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest. We address ourselves, not to their humanity but to their self-love, and never talk to them of our necessities but of their advantages. —The Wealth Of Nations, Book I

As an organization ages and grows, its founders retire or move on, it becomes more dependent on policies and regulations and each individual employee finds his own incentives further displaced from the company’s original intentions. Soon a company is no longer devoted to either the well-being of its founders or its customers, but to the company itself. (And that’s kind of a best-case scenario in which the company doesn’t just disintegrate into individual self-interest.)

I am reminded of a story about a computer that had been programmed to play Tetris–actually, it had been programmed not to lose at Tetris. So the computer paused the game. A paused game cannot lose.

What percentage of employees (especially management) have been incentivized to win? And what percentage are being incentivized to not lose?

And no, I don’t mean that in some 80s buzzword-esque way. Most employees have more to lose (ie, their jobs) if something goes wrong as a result of their actions than to gain if something goes right. The stockholders might hope that employees are doing everything they can to maximize profits, but really, most people are trying not to mess up and get fired.

Fear of messing up goes beyond the individual scale. Whole companies are goaded by concerns about risk–“Could we get sued?” Large corporation have entire legal teams devoted to telling them how they could get sued for whatever their doing and to filing lawsuits against their competitors for whatever they’re doing.

This fear of risk carries over, in turn, to government regulations. As John Sanphillipo writes in City Regulatory Hurdles Favor Big Developers, not the Little Guy:

A family in a town I visited bought an old fire station a few years ago with the intention of turning it into a Portuguese bakery and brewpub. They thought they’d have to retrofit the interior of the building to meet health and safety standards for such an establishment.

Turns out the cost of bringing the landscape around the outside of the building up to code was their primary impediment. Mandatory parking requirements, sidewalks, curb cuts, fire lanes, on-site stormwater management, handicapped accessibility, drought-tolerant native plantings…it’s a very long list that totaled $340,000 worth of work. … Guess what? They decided not to open the bakery or brewery. …

Individually it’s impossible to argue against each of the particulars. Do you really want to deprive people in wheelchairs of the basic civil right of public accommodation? Do you really want the place to catch fire and burn? Do you want a barren landscape that’s bereft of vegetation? …

I was in Hamtramck, Michigan a couple of years ago to participate in a seminar about reactivating neighborhoods through incremental small-scale development. …

While the event was underway the fire marshal happened to drive by and noticed there were people—a few dozen actual humans—occupying a commercial building in broad daylight. In a town that has seen decades of depopulation and disinvestment, this was an odd sight. And he was worried. Do people have permission for this kind of activity? Had there been an inspection? Was a permit issued? Is everything insured? He called one of his superiors to see if he should shut things down in the name of public safety.

It’s a good article. You should read the whole thing.

Back in Phillipe Bourgeois’s In Search of Respect: Selling Crack in el Barrio, Phillipe describes one drug dealer’s attempt to use the money he’d made to go into honest business by opening a convenience store. Unfortunately, he couldn’t get the store complaint with NYC disability-access regulations, and so the store never opened and the owner went back to dealing drugs. (What IQ, I wonder, is necessary to comply with all of these laws and regulations in the first place?)

Now, I’m definitely in favor of disabled people being able to buy groceries and use bathrooms. But what benefits a disabled person more: a convenience store that’s not fully wheel-chair accessible, or a crack house?

In My IRB Nightmare, Scott Alexander writes about trying to do a simple study to determine whether the screening test already being used to diagnose people with bipolar disorder is effective at diagnosing them:

When we got patients, I would give them the bipolar screening exam and record the results. Then Dr. W. would conduct a full clinical interview and formally assess them. We’d compare notes and see how often the screening test results matched Dr. W’s expert diagnosis.

Remember, they were already using the screening test on patients and then having them talk to the doctor for a formal assessment. The only thing the study added was that Scott would compare how well the screening results matched the formal assessment. No patients would be injected, subject to new procedures, or even asked different questions. They just wanted to compare two data sets.

After absurd quantities of paperwork and an approval process much too long to summarize here, the project got audited:

I kept the audit report as a souvenier. I have it in front of me now. Here’s an example infraction:

The data and safety monitoring plan consists of ‘the Principal Investigator will randomly check data integrity’. This is a prospective study with a vulnerable group (mental illness, likely to have diminished capacity, likely to be low income) and, as such, would warrant a more rigorous monitoring plan than what is stated above. In addition to the above, a more adequate plan for this study would also include review of the protocol at regular intervals, on-going checking of any participant complaints or difficulties with the study, monitoring that the approved data variables are the only ones being collected, regular study team meetings to discuss progress and any deviations or unexpected problems. Team meetings help to assure participant protections, adherence to the protocol. Having an adequate monitoring plan is a federal requirement for the approval of a study. See Regulation 45 CFR 46.111 Criteria For IRB Approval Of Research. IRB Policy: PI Qualifications And Responsibility In Conducting Research. Please revise the protocol via a protocol revision request form. Recommend that periodic meetings with the research team occur and be documented.

… Faced with submitting twenty-seven new pieces of paperwork to correct our twenty-seven infractions, Dr. W and I gave up. We shredded the patient data and the Secret Code Log. We told all the newbies they could give up and go home. … We told the IRB that they had won, fair and square; we surrendered unconditionally.

The point of all that paperwork and supervision is to make sure that no one replicates the Tuskegee Syphilis Experiment nor the Nazi anything. Noble sentiments–but as a result, a study comparing two data sets had to be canceled.

I’ve noticed recently that much of the interesting medical research is happening in the third world/China–places where the regulations aren’t as strong and experiments (of questionable ethics or not) can actually get done.

Like the computer taught not to lose at Tetris, all of these systems are more focused on minimizing risk–even non-existent risk–than on actually succeeding.

In his review of Yudkowsky’s Inadequate Equilibria, Scott writes:

…[Yudkowsky] continues to the case of infant parenteral nutrition. Some babies have malformed digestive systems and need to have nutrient fluid pumped directly into their veins. The nutrient fluid formula used in the US has the wrong kinds of lipids in it, and about a third of babies who get it die of brain or liver damage. We’ve known for decades that the nutrient fluid formula has the wrong kind of lipids. We know the right kind of lipids and they’re incredibly cheap and there is no reason at all that we couldn’t put them in the nutrient fluid formula. We’ve done a bunch of studies showing that when babies get the right nutrient fluid formula, the 33% death rate disappears. But the only FDA-approved nutrient fluid formula is the one with the wrong lipids, so we just keep giving it to babies, and they just keep dying. Grant that the FDA is terrible and ruins everything, but over several decades of knowing about this problem and watching the dead babies pile up, shouldn’t somebody have done something to make this system work better?

The doctors have to use the FDA-approved formula or they could get sued for malpractice. The insurance companies, of course, only cover the FDA-approved formula. The formula makers are already making money selling the current formula and would probably have to go through an expensive, multi-year review system (with experiments far more regulated than Scott’s) to get the new formula approved, and even then they might not actually get approval. In short, on one side are people in official positions of power whose lives could be made worse (or less convenient) if they tried to fix the problem, and on the other side are dead babies who can’t stand up for themselves.

The Chankiri Tree (Killing Tree) where infants were fatally smashed, Choeung Ek, Cambodia.

Communism strikes me as the ultimate expression of this beast: a society fully transformed into a malevolent AI. It’s impossible to determine exactly how many people were murdered by communism, but the Black Book of Communism estimates a death toll between 85 and 100 million people.

Capitalism, for all its faults, is at least somewhat decentralized. If you make a bad business decision, you suffer the consequences and can hopefully learn from your mistakes and make better decisions in the future. But in communist systems, one central planner’s bad decisions can cause suffering for millions of other people, resulting in mass death. Meanwhile, the central planner may suffer for correcting the bad decision. Centralized economies simply lack the feedback loops necessary to fix problems before they start killing people.

While FDA oversight of medicines is probably important, would it be such a bad thing if a slightly freer market in parenteral nutrition allowed parents to chose between competing brands of formula, each promising not to kill your baby?

Of course, capitalism isn’t perfect, either. SpottedToad recently had an interesting post, 2010s Identity Politics as Hostile AI:

There’s an interesting post mortem on the rise and fall of the clickbait liberalism site Mic.com, that attracted an alleged 65 million unique visitors on the strength of Woketastic personal stories like “5 Powerful Reasons I’m a (Male) Feminist,” …

Every time Mic had a hit, it would distill that success into a formula and then replicate it until it was dead. Successful “frameworks,” or headlines, that went through this process included “Science Proves TK,” “In One Perfect Tweet TK,” “TK Reveals the One Brutal Truth About TK,” and “TK Celebrity Just Said TK Thing About TK Issue. Here’s why that’s important.” At one point, according to an early staffer who has since left, news writers had to follow a formula with bolded sections, which ensured their stories didn’t leave readers with any questions: The intro. The problem. The context. The takeaway.

…But the success of Mic.com was due to algorithms built on top of algorithms. Facebook targets which links are visible to users based on complex and opaque rules, so it wasn’t just the character of the 2010s American population that was receptive to Mic.com’s specific brand of SJW outrage clickbait, but Facebook’s rules for which articles to share with which users and when. These rules, in turn, are calibrated to keep users engaged in Facebook as much as possible and provide the largest and most receptive audience for its advertisers, as befits a modern tech giant in a two-sided market.

Professor Bruce Charlton has a post about Head Girl Syndrome–the Opposite of Creative Genius that is good and short enough that I wish I could quote the whole thing. A piece must suffice:

The ideal Head Girl is an all-rounder: performs extremely well in all school subjects and has a very high Grade Point Average. She is excellent at sports, Captaining all the major teams. She is also pretty, popular, sociable and well-behaved.

The Head Girl will probably be a big success in life, in whatever terms being a big success happens to be framed …

But the Head Girl is not, cannot be, a creative genius. …

The more selective the social system, the more it will tend to privilege the Head Girl and eliminate the creative genius.

Committees, peer review processes, voting – anything which requires interpersonal agreement and consensus – will favour the Head Girl and exclude the creative genius.  …

*

We live in a Head Girl’s world – which is also a world where creative genius is marginalized and disempowered to the point of near-complete invisibility.

The quest for social status is, I suspect, one of the things driving the system. Status-oriented people refuse to accept information that comes from people lower status than themselves, which renders system feedback even more difficult. The internet as a medium of information sharing is beautiful; the internet as a medium of status signalling is horrible.

So what do you think? Do sufficiently large organization start acting like malevolent (or hostile) AIs?

(Back to Part 1)