Do Sufficiently Large Organizations Start Acting Like Malevolent AIs? (pt 2)

(Part 1 is here)

As we were discussing on Monday, as our networks have become more effective, our ability to incorporate new information may have actually gone down. Ironically, as we add more people to a group–beyond a certain limit–it becomes more difficult for individuals with particular expertise to convince everyone else in the group that the group’s majority consensus is wrong.

The difficulties large groups experience trying to coordinate and share information force them to become dominated by procedures–set rules of behavior and operation are necessary for large groups to operate. A group of three people can use ad-hoc consensus and rock-paper-scissors to make decisions; a nation of 320 million requires a complex body of laws and regulations. (I once tried to figure out just how many laws and regulations America has. The answer I found was that no one knows.)

An organization is initially founded to accomplish some purpose that benefits its founders–generally to make them well-off, but often also to produce some useful good or service. A small organization is lean, efficient, and generally exemplifies the ideals put forth in Adam Smith’s invisible hand:

It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest. We address ourselves, not to their humanity but to their self-love, and never talk to them of our necessities but of their advantages. —The Wealth Of Nations, Book I

As an organization ages and grows, its founders retire or move on, it becomes more dependent on policies and regulations and each individual employee finds his own incentives further displaced from the company’s original intentions. Soon a company is no longer devoted to either the well-being of its founders or its customers, but to the company itself. (And that’s kind of a best-case scenario in which the company doesn’t just disintegrate into individual self-interest.)

I am reminded of a story about a computer that had been programmed to play Tetris–actually, it had been programmed not to lose at Tetris. So the computer paused the game. A paused game cannot lose.

What percentage of employees (especially management) have been incentivized to win? And what percentage are being incentivized to not lose?

And no, I don’t mean that in some 80s buzzword-esque way. Most employees have more to lose (ie, their jobs) if something goes wrong as a result of their actions than to gain if something goes right. The stockholders might hope that employees are doing everything they can to maximize profits, but really, most people are trying not to mess up and get fired.

Fear of messing up goes beyond the individual scale. Whole companies are goaded by concerns about risk–“Could we get sued?” Large corporation have entire legal teams devoted to telling them how they could get sued for whatever their doing and to filing lawsuits against their competitors for whatever they’re doing.

This fear of risk carries over, in turn, to government regulations. As John Sanphillipo writes in City Regulatory Hurdles Favor Big Developers, not the Little Guy:

A family in a town I visited bought an old fire station a few years ago with the intention of turning it into a Portuguese bakery and brewpub. They thought they’d have to retrofit the interior of the building to meet health and safety standards for such an establishment.

Turns out the cost of bringing the landscape around the outside of the building up to code was their primary impediment. Mandatory parking requirements, sidewalks, curb cuts, fire lanes, on-site stormwater management, handicapped accessibility, drought-tolerant native plantings…it’s a very long list that totaled $340,000 worth of work. … Guess what? They decided not to open the bakery or brewery. …

Individually it’s impossible to argue against each of the particulars. Do you really want to deprive people in wheelchairs of the basic civil right of public accommodation? Do you really want the place to catch fire and burn? Do you want a barren landscape that’s bereft of vegetation? …

I was in Hamtramck, Michigan a couple of years ago to participate in a seminar about reactivating neighborhoods through incremental small-scale development. …

While the event was underway the fire marshal happened to drive by and noticed there were people—a few dozen actual humans—occupying a commercial building in broad daylight. In a town that has seen decades of depopulation and disinvestment, this was an odd sight. And he was worried. Do people have permission for this kind of activity? Had there been an inspection? Was a permit issued? Is everything insured? He called one of his superiors to see if he should shut things down in the name of public safety.

It’s a good article. You should read the whole thing.

Back in Phillipe Bourgeois’s In Search of Respect: Selling Crack in el Barrio, Phillipe describes one drug dealer’s attempt to use the money he’d made to go into honest business by opening a convenience store. Unfortunately, he couldn’t get the store complaint with NYC disability-access regulations, and so the store never opened and the owner went back to dealing drugs. (What IQ, I wonder, is necessary to comply with all of these laws and regulations in the first place?)

Now, I’m definitely in favor of disabled people being able to buy groceries and use bathrooms. But what benefits a disabled person more: a convenience store that’s not fully wheel-chair accessible, or a crack house?

In My IRB Nightmare, Scott Alexander writes about trying to do a simple study to determine whether the screening test already being used to diagnose people with bipolar disorder is effective at diagnosing them:

When we got patients, I would give them the bipolar screening exam and record the results. Then Dr. W. would conduct a full clinical interview and formally assess them. We’d compare notes and see how often the screening test results matched Dr. W’s expert diagnosis.

Remember, they were already using the screening test on patients and then having them talk to the doctor for a formal assessment. The only thing the study added was that Scott would compare how well the screening results matched the formal assessment. No patients would be injected, subject to new procedures, or even asked different questions. They just wanted to compare two data sets.

After absurd quantities of paperwork and an approval process much too long to summarize here, the project got audited:

I kept the audit report as a souvenier. I have it in front of me now. Here’s an example infraction:

The data and safety monitoring plan consists of ‘the Principal Investigator will randomly check data integrity’. This is a prospective study with a vulnerable group (mental illness, likely to have diminished capacity, likely to be low income) and, as such, would warrant a more rigorous monitoring plan than what is stated above. In addition to the above, a more adequate plan for this study would also include review of the protocol at regular intervals, on-going checking of any participant complaints or difficulties with the study, monitoring that the approved data variables are the only ones being collected, regular study team meetings to discuss progress and any deviations or unexpected problems. Team meetings help to assure participant protections, adherence to the protocol. Having an adequate monitoring plan is a federal requirement for the approval of a study. See Regulation 45 CFR 46.111 Criteria For IRB Approval Of Research. IRB Policy: PI Qualifications And Responsibility In Conducting Research. Please revise the protocol via a protocol revision request form. Recommend that periodic meetings with the research team occur and be documented.

… Faced with submitting twenty-seven new pieces of paperwork to correct our twenty-seven infractions, Dr. W and I gave up. We shredded the patient data and the Secret Code Log. We told all the newbies they could give up and go home. … We told the IRB that they had won, fair and square; we surrendered unconditionally.

The point of all that paperwork and supervision is to make sure that no one replicates the Tuskegee Syphilis Experiment nor the Nazi anything. Noble sentiments–but as a result, a study comparing two data sets had to be canceled.

I’ve noticed recently that much of the interesting medical research is happening in the third world/China–places where the regulations aren’t as strong and experiments (of questionable ethics or not) can actually get done.

Like the computer taught not to lose at Tetris, all of these systems are more focused on minimizing risk–even non-existent risk–than on actually succeeding.

In his review of Yudkowsky’s Inadequate Equilibria, Scott writes:

…[Yudkowsky] continues to the case of infant parenteral nutrition. Some babies have malformed digestive systems and need to have nutrient fluid pumped directly into their veins. The nutrient fluid formula used in the US has the wrong kinds of lipids in it, and about a third of babies who get it die of brain or liver damage. We’ve known for decades that the nutrient fluid formula has the wrong kind of lipids. We know the right kind of lipids and they’re incredibly cheap and there is no reason at all that we couldn’t put them in the nutrient fluid formula. We’ve done a bunch of studies showing that when babies get the right nutrient fluid formula, the 33% death rate disappears. But the only FDA-approved nutrient fluid formula is the one with the wrong lipids, so we just keep giving it to babies, and they just keep dying. Grant that the FDA is terrible and ruins everything, but over several decades of knowing about this problem and watching the dead babies pile up, shouldn’t somebody have done something to make this system work better?

The doctors have to use the FDA-approved formula or they could get sued for malpractice. The insurance companies, of course, only cover the FDA-approved formula. The formula makers are already making money selling the current formula and would probably have to go through an expensive, multi-year review system (with experiments far more regulated than Scott’s) to get the new formula approved, and even then they might not actually get approval. In short, on one side are people in official positions of power whose lives could be made worse (or less convenient) if they tried to fix the problem, and on the other side are dead babies who can’t stand up for themselves.

The Chankiri Tree (Killing Tree) where infants were fatally smashed, Choeung Ek, Cambodia.

Communism strikes me as the ultimate expression of this beast: a society fully transformed into a malevolent AI. It’s impossible to determine exactly how many people were murdered by communism, but the Black Book of Communism estimates a death toll between 85 and 100 million people.

Capitalism, for all its faults, is at least somewhat decentralized. If you make a bad business decision, you suffer the consequences and can hopefully learn from your mistakes and make better decisions in the future. But in communist systems, one central planner’s bad decisions can cause suffering for millions of other people, resulting in mass death. Meanwhile, the central planner may suffer for correcting the bad decision. Centralized economies simply lack the feedback loops necessary to fix problems before they start killing people.

While FDA oversight of medicines is probably important, would it be such a bad thing if a slightly freer market in parenteral nutrition allowed parents to chose between competing brands of formula, each promising not to kill your baby?

Of course, capitalism isn’t perfect, either. SpottedToad recently had an interesting post, 2010s Identity Politics as Hostile AI:

There’s an interesting post mortem on the rise and fall of the clickbait liberalism site, that attracted an alleged 65 million unique visitors on the strength of Woketastic personal stories like “5 Powerful Reasons I’m a (Male) Feminist,” …

Every time Mic had a hit, it would distill that success into a formula and then replicate it until it was dead. Successful “frameworks,” or headlines, that went through this process included “Science Proves TK,” “In One Perfect Tweet TK,” “TK Reveals the One Brutal Truth About TK,” and “TK Celebrity Just Said TK Thing About TK Issue. Here’s why that’s important.” At one point, according to an early staffer who has since left, news writers had to follow a formula with bolded sections, which ensured their stories didn’t leave readers with any questions: The intro. The problem. The context. The takeaway.

…But the success of was due to algorithms built on top of algorithms. Facebook targets which links are visible to users based on complex and opaque rules, so it wasn’t just the character of the 2010s American population that was receptive to’s specific brand of SJW outrage clickbait, but Facebook’s rules for which articles to share with which users and when. These rules, in turn, are calibrated to keep users engaged in Facebook as much as possible and provide the largest and most receptive audience for its advertisers, as befits a modern tech giant in a two-sided market.

Professor Bruce Charlton has a post about Head Girl Syndrome–the Opposite of Creative Genius that is good and short enough that I wish I could quote the whole thing. A piece must suffice:

The ideal Head Girl is an all-rounder: performs extremely well in all school subjects and has a very high Grade Point Average. She is excellent at sports, Captaining all the major teams. She is also pretty, popular, sociable and well-behaved.

The Head Girl will probably be a big success in life, in whatever terms being a big success happens to be framed …

But the Head Girl is not, cannot be, a creative genius. …

The more selective the social system, the more it will tend to privilege the Head Girl and eliminate the creative genius.

Committees, peer review processes, voting – anything which requires interpersonal agreement and consensus – will favour the Head Girl and exclude the creative genius.  …


We live in a Head Girl’s world – which is also a world where creative genius is marginalized and disempowered to the point of near-complete invisibility.

The quest for social status is, I suspect, one of the things driving the system. Status-oriented people refuse to accept information that comes from people lower status than themselves, which renders system feedback even more difficult. The internet as a medium of information sharing is beautiful; the internet as a medium of status signalling is horrible.

So what do you think? Do sufficiently large organization start acting like malevolent (or hostile) AIs?

(Back to Part 1)


18 thoughts on “Do Sufficiently Large Organizations Start Acting Like Malevolent AIs? (pt 2)

  1. What a great post. I have often pondered about the sheer mass of people and ideas and how that might work against any real possibility of governing, or at least a democratic governing. And I have also ponder the idea that once a certain threshold of population is reached that really we just maintain an idea of democracy but what occurs is that there is some sort of upper strata that functions essentially independent from the larger ideal of the democratic society. In short, the aggregate of people will still have to believe that there is a functioning democratic society and indeed behave in various democratic activist socially responsible manners. But the actual functioning of that state will be allowed by a certain segment that pretty much remains in visible to that democratic religious ideal.

    These two sectors will not reduce to one another. I figure this means that the functioning of the democratic society, as it indeed is functioning as a large group of people having a voice, Will in essence and in all practicality, be cut off from the effective group by which that democracy functions. By virtue of the Democratic faith of the people that are invested in kind of so to speak the lower echelon’s of the functioningsociety, they will be completely unable to conceptualize what is actually allowing for their ideology to manifest and be maintained. I have speculated that this upper echelon, for lack of a better term, will be for all effective reasoning, and authoritarian oligarchy basically. But there will be no conceptual course of the “democratic people quote, the “democratic society” to effectively check this upper Strata. And this is to say that through all their democratic behaviour they will indeed see their activity as affecting this upper strata of mechanistic totalitarianism, but the fact of the matter is that it will be unaffected. It will behave in the manner that it behaves despite what the democratic ideal would want to assume that it imposes upon it.

    Any fact, what I’m describing sounds very similar to your post here about a I.

    What do you think?

    ( I hope the auto correct on my voice dictation didn’t read that reply totally meaningless. Lol).


    • I’m sorry. I wrote you a replay and then accidentally deleted it. The summary version was: 1. Thank you! I’m glad you liked the post. I wasn’t sure about it and so sat on it for about 6 months. (It still feels rather disjointed to me, but at some point it has to be done.)

      I think what you’re describing is essentially a “shadow government” or deep state, which of course I believe in. The Deep State is largely Democrat but has plenty of defense contractors who benefited from Bush II. The deep state does most of the day-to-do running of the country and has most of the real power (if anyone does) over people’s lives. The Deep State recognizes career politicians, no matter how slimy or immoral, as its allies–they fund it and pass laws that give it more power. Trump is interesting because the Deep State does not recognize him and so is doing everything it can to reject him, like a body rejecting an incompatible liver transplant.

      (The same thing, we could argue, happened to Nixon.)

      Contrast Trump with Bush II. Trump gets called racist or accused of hating Muslims all the time, but Bush actually started a war on phony pretexts that cost trillions of dollars and killed thousands of Muslims (and Americans.) He had popular support from Republicans, from Congress, from Democrats, from the media, etc. In his first election IIRC he received endorsements from about half the papers.

      To this day we have yet to have a major reckoning from the government or media about what a collosal, irresponsible, tragic waste of human lives (and money) this war was. They haven’t apologized or promised to do anything to ensure that next time, thousands of people don’t get killed in a fabricated war.

      By contrast, Trump got something like two newspaper endorsements. Two. The same people who were fine with murdering thousands of Americans and Iraqis think Trump is beyond the pale. Bush, for all his faults, was at least a clear friend of the Deep State–Washington DC and its suburbs flourished during his term as those “Homeland Security” dollars rolled in. Trump, by contrast, they clearly regard as a threat (whether he is or not–the Deep State is pretty powerful, after all.)

      Thanks for your thoughtful comment; as far as I can tell, the voice dictation worked perfectly.


  2. … The bad decisions, conversely, of the “umbrella oligarchy“ or the “AI“, Will be understood inherently and automatically as a situation that can be remedied through the democratic process. But indeed at every turn such mistaken rulings of the sort of oligarchy will be an acted upon the Democratic population regardless of what voice they are believing in truth to be having. The Democratic subject will see axiomatically it’s behaviour as affecting the whole in a sensible manner. But it will be more like that movie “Stutter Island” with Leonardo DiCaprio. The Democratic capitalist subject will behave in a matter that is self reflective of its own understanding of what’s occurring. But in fact there will be a segment of the population that is completely immune and in fact understands what this democratic subject is doing why and how they are behaving in a manner they behave, and so will be able to act in a manner that while ultimately controlling the Democratic subject, likewise allows that subject to behave in a manner that she sees as free choice.


  3. Terry Gilliam’s “Brazil” comes to mind reading your (awesome) post. It seems like those people standing behind internet (or rather, behind the monopolies of internet, i.e. Google and the likes) hit the mother lode. They are gaining control over the majority of population by the minute; a control much more powerful than any government can dream of. I don’t even know how to begin to explain to even a very smart teenager or young adult (the usual targets) how all that stuff—the smartphones, the social media, the availability of just about everything they can think of—is actually NOT cool. Being in their age and position, I would probably also think it was bloody cool. I also don’t know where to go to get away from it all.


    • Thanks.
      My own option (which seems to work for me) is simply not to use a smartphone. I don’t use the internet, Facebook, Twitter, or anything else on my phone. It’s just for calling people. So when I’m out and about, I’m not checking my phone. i don’t know how things are going on Twitter or what the latest outrage is. Of course, I can check whatever I want at home, but my computer is going on 11 years, so that’s limited, too.

      We don’t need 24/7 connectivity. We need to take the occasional break, to step back, take a few deep breaths and ask ourselves, “does this really matter?”


      • I hear ya, it’s pretty much the same for me, including a 8 years old computer. What works works. I noticed that the usual “there-are-two-kinds-of-people” applies to this situation as well. There are people who constantly follow a lot of different sites, news, Twitter, Facebook, YouTube channels, all kinds of different feeds. Depending on the level of their mental fitness, they either sort of passively take in snippets of that enormous information,
        or consciously brute force as much cold data in their memory as possible (

        The other kind of people close out most of this incredible noise, and prefer to actively search for (often very specific) things, filter out what’s not needed and select only what they’re truly interested in.


  4. This post is BIG. It opens up a whole can of worms.The idea that organizations lose focus has been summarized by a great Man and recently deceased, Jerry Pournelle, as “The Iron Law of Bureaucracy”.

    “Pournelle’s Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people”:

    First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration.

    Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc.

    The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.”

    I also have another theory. I believe that over time, we’re talking hundreds of years. any country will eventually have psychopaths move to control the heights of a country. We all know they exist and we all know they lust for power over us but it hard to believe that these people are the super nice, perfect actors, who run things, but they are. This is a totally separate issue from the bureaucracy. The bureaucracy helps them hide as no one is really responsible for anything. The bigger the organization and the longer it’s been around the more likely it is to be run by or compromised by psychopaths.

    Our country had a way of dealing with this. Separate issues into different levels of government and let local government deal with most issues. This of course wasn’t perfect and the busy bodies refused to accept anything different so now it all screwed up and no one can make it work. It’s possible, frankly most likely, that it will continue to get worse and the only way to make it better will be the tear the whole structure down and destroy it completely. I’m not looking forward to this or wishing it happen but that seems to be the common response throughout history. Look at this population of Rome chart.

    After it was sacked everyone just left. It became a empty city. The whole edifice completely collapsed. They had much the same bureaucracy. You couldn’t even change work. You had to be what your father was. It became absurd.

    Here’s another astounding example This is the Temple of Hera. One of my FAVORITE buildings. A bunch of workmen were building a road and found two huge temples from around 500B.C. in the woods that had been abandoned since the middle ages!

    Let’s hope this doesn’t happen to us.


    • The second group sounds like what I call lizard people. They love running organizations for no discernible purpose besides enforcing rules; they seem to have no awareness that the function of the organization they’re running is supposed to be to make people’s lives better in some manner. I think they’re attuned to social status, but not much else.

      I often imagine what future archaeologists in a thousand or ten thousand years will make of our cities and monuments. What will survive? Will they remember us? Like so many peoples named after the pots they made, will we someday be known only by our cultural artifacts? (“The Golden Arches People”)


      • Actually our whole civilization will crumble to dust very fast. The reason is concrete is strengthened by steel bars(rebar), used to increase the tensile strength of concrete. All concrete has micro-cracks and water gets in, rust the bars, which opens the cracks bigger, til the whole thing crumbles to dust and the iron bars corrode away.

        They have new reinforcement made of glass fibers. Some are regular glass, some e-glass(stronger) and my favorite basalt fiber glass made from basalt volcano glass.

        With these you could build roads, bridges and building that would last almost forever. We should make it mandatory that all reinforcement in roads and bridges is made of this stuff. Right now the concrete starts having problems in about 50 years. The main reason our roads and bridges are falling down.


  5. Well done Evolution Theorist. I would add to your article the following points:

    a) Many companies have so low profit margins, or their income statements are long time in the red, that profit obviously is not these organizations main motive. They provide work and salaries for their employees and managers, and this is their main motive. This can be equally well be the case in companies which produce large profits. Organization has to survive and function first before it can produce the services or products it produces, so existing often wins out producing. It is assumed that the main motive of upholding large public bureaucracies is to produce what they produce, but this can largely be subverted by the work and salary motives of employees and managers, because public bureaucracies are not disciplined by the market. In the fuzzy opaqueness of unclear cost benefit ratio bureaucracy can grow very turgid.

    b) Psychological biases strengthen the effects you described:

    1) My side bias and ingroup bias (or my organization bias): “My organization is good, better and moral. In disputes with outsiders I support in principle and without much thought my organization.” Person sees his organization as an extension of himself, and thinks, does and reacts accordingly.

    2) Cognitive dissonance and its reduction: “I am good person and my organization is also good – my organization produces bad outcomes for people —> people must be bad, lazy, unjust, stupid and dont follow the rules, thus they have caused the bad outcomes to themselves —> me and my organization are still good.”

    3) Social conformity: “I conform to socially accepted in my organization, and dont question or resist much or at all.”

    4) Tendency to believe authorities in the organization: “If they give biased, false, distorted, exaggerated or lacking information, I am still inclined to believe it.”

    5) Just world bias: “Did those bad outcomes happen to them because they were bad entrepreneurs or because regulations burdened them too much? I have disposition to believe in just world, so they must have done something bad or insufficient to deserve what they got.”

    6) Identifiable victim bias: “Touching personal story of a suffering person tends to be more important to me than statistics about 10000 suffering persons.” Organizations rely mostly on statistics.

    7) System justification bias: “I have tendency to believe that the present social, economic, political and societal arrangements are better than the alternatives on offer, if not the best. I have propensity to devalue and disparage the alternatives.”

    8) Naive realism: “I have predisposition to believe that I see the world as it is, there is not much or anything to correct in or add to my understanding of the world.”



    • Thank you. You make some great points. I am especially reminded of some lawyers I’ve known who really seemed to believe the cases they were prosecuting were just even though, from the outside, the whole process just looked like a terrible waste of time and money with no justice involved at all. Lawyers tend to do a better job if they believe themselves to be on the right side, legally speaking–but they can’t both be. In the end they make a lot of money, and everyone else is worse off.


  6. “Status-oriented people refuse to accept information that comes from people lower status than themselves”

    You accidentally explained the demand for the consulting industry.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s