Book review: The Unaccountability Machine¶

I’ve been interested in why organizations work (or seem to not work, despite everyone in them being good people). Another book cited the book “The Unaccountability machine” (wikipedia, HelMet libraries) as being related to this topic, so I checked it out. In the end, the book did provide some thoughts on this, mainly through the lens of complex webs of decision making and accountability sinks. The first half was most relevant to me; the second half ventured into economics and society (which was still interesting, but not my main focus).

Summary¶

In modern times, the word “cybernetics” makes one think of human/machine combinations or maybe computer technology in general, but the original definition (1940s and onward) was about decision making, feedback, and control of systems (machines, organizations, economies, etc.). It’s important to keep this in mind, otherwise the topic and review below can easily get confused with unrelated technology matters. Cybernetics and the book are about information flows.

Systems don’t have motivations. They are made up of people who have motivations and outside influences. How can a system made of good people have bad outcomes? How can it be avoided? These are the kinds of questions I wanted to answer.

There are several main lessons I got from the book:

The number of states a system can be in vs the amount possibly gradations of control input. This is known as “variety”. An example from my university:

The university defines there are four categories of data (levels of confidentiality), and that should dictate how the data is handled. However, a university is a complex thing with very different {operations, teaching, research} divisions, and four categories to control data (input variety) just isn’t enough to express the wide variety of actual data (and there isn’t enough variety of other processes or exceptions). This leads to people doing whatever they want (ignoring the rules since they don’t seem applicable), or never-ending discussions about how the system needs to be tweaked to make it work.

The idea that the internal complexity of a system has to match the external complexity it faces is the Law of Requisite Complexity, and once I realized this, I started seeing almost every dysfunctional thing of our university being caused by its violation.
Cyberneticists have broken down organizations into five systems needed for their control (Viable system model) (operations, regulation, optimization or integration, intelligence, and philosophy or identity, see more in Chapter 5’s summary below). My own team is too small to have these as defined roles, but thinking of them helps me realize some things we need to make sure get done.
Any complex organization isn’t one mind, or many individuals, but many different parts (people) communicating in a complex network. This can easily allow decisions to be made and propagated elsewhere in a way that doesn’t quite make sense, with little feedback or accountability to the place the decisions were made. This makes the accountability sinks (which was apparently defined here, though I would have thought it was older).

A complex network of nodes sending signals to each other, sometimes having a result that is undesired? Sounds like “artificial intelligence” these days, with hallucinations and wrong answers, doesn’t it? Just like with “AI”, there needs to be a feedback system to keep an eye on things.

The second half of the book is focused on economics and society (because the author is somewhat of an economist) and less relevant to my work, but I found that it was quite useful to me anyway. Overall, I found some good points that I could apply to my own non-economics life.

Overall thoughts¶

I liked the book and was happy to read something that combined something in my comfort zone (science-ish description of cybernetics) with something outside it (economics). I’ve learned some buzzwords that I can use in my job to push for changes, though I predict it’s an uphill battle to get anything actually done.

As my team gets more complex, we need to encapsulate different parts and have their internal variety and management, while having a more limited variety that reaches higher levels for the systematic decision making.

Chapter-by-chapter analysis¶

Below is a summary of each chapter. I state what the book says (with some of my opinions or take-away messages mixed in), but I am not qualified to judge if the book is correct or not.

Part 1: The nature of the crisis¶

(0) Introduction¶

The industrialization of society makes larger and more complex organizations, and they need to be controlled. The theory of these control structures is “cybernetics” (the original definition).

I liked this intro, because it related to my thing about industrialization (of finance, control, etc) being the more relevant problem in our society than just capitalism that most people complain about. The idea is that when things get too big, they somehow makes more than their share of problems.

(1) Something’s up¶

Organizations sometimes do things that just make no sense, for example, the banking crisis of 2008. No person wanted this, yet somehow it happened. There were even people who could predict it, but no one would listen to them because the system was on autopilot and not taking feedback.

Accountability sinks are ways that decisions can be made such that no individual person is accountable. A decision may come out of the organization, but all the individuals of the organization can say “I don’t agree with this but it’s not me who decided” - so where does it come from? Who takes accountability? These are quite common when you start looking for them, when everyone can pass off the responsibility for a decision/rule to someone else (often laundering it through various policies or laws).

I will choose to not mention one example the book gives involving animals, since it’s just too tragic. Another example given is academic publishing. Professional publishers have managed to insert themselves into the academic system and extract huge amounts of money, and most people agree there are serious problems with relying almost exclusively on citations for academic career purposes. Yet, still, it stays, because it’s now part of the system and the politics of making any other decision about careers would be too much.

There are two types of decision-makers, kings (who are humans and to whom decisions can be appealed), and priests (who speak the word of god and thus you can’t appeal to them). Too often, a decision may be made by a “king” but gets conveyed by “priests”, and there is no way to send feedback that something is wrong. This can cause unrest.

All the above isn’t just an accident. It’s a trade-off, discretion (to do the right thing) vs following the rules. What’s the balance? This is the next chapter.

Overall, this chapter said what I wanted to hear about organizations, bad choices, and and unaccountability and left me wanting an analysis.

(2) Stafford Beer¶

This chapter is about the historical context.

Stafford Beer was one of the founders of management cybernetics - as discussed above, how decisions get made. Beer looked at systems as composed of black boxes (something with defined inputs and outputs, but the internals can’t be examined): different components, and each component had some behavior. You couldn’t/didn’t look into the boxes. Of course, you could recursively look into them if you wanted, or package multiple black boxes into another black box. Still, at some level, one has to draw black boxes around some parts of the system and analyze the parts outside of that.

Management cyberneticists looked at systems as black boxes and that interact with each other, to discover various theories of control. Thinking in terms of black boxes gives the saying “the purpose of a system is what it does”. This is important to keep in mind so that one doesn’t blame the individuals for what the system does - you have to look at the connections and control structures.

In the end, systems/organizations are controlled by meetings between the different levels and components. This often happens by resource bargains between different components, so that each part can focus on its internals with a defined interface to the outside.

I understand the concept of the black boxes, it took some time to fully internalize the recursive nature of them. I’m still not quite clear where one should decide to draw the black boxes (what level and what size), but I guess that’s part of the art of the field. You should probably draw them at the places that are most useful for your current problem. In the end, this question doesn’t matter too much for my understanding.

(3) Aliens among us¶

So, a system is made up of black boxes once you go deep enough. These are all connected and sending signals to each other. It becomes a… neural network? There are clear metaphors to “artificial intelligence” here, where the system takes on a mind of its own, but each individual part can’t have accountability. This also relates to the Chinese room (translation) thought experiment.

This applies to all kinds of things, from social media algorithms to corporate management. Yet, we know the organization itself is not sentient. So, the organization itself isn’t accountable, but neither are the independent parts (because they are just playing their role).

The more connections a system has, the harder it is to understand. However hard you look, you will probably miss something.

If you are interacting with one of these systems, you can be given the policies/documents in response to a query about why a decision was made a certain way. But that never explains why the policies are made a certain way. The accountability for them has been sunk into the network itself, and if people only try to hold individuals accountable, everyone can escape accountability.

So, in the end, you see the same thing over and over again. Comments about an organization that say they admire all the people inside, but despise the overall organization.

This part of organizations being a computing network really made sense to me and started to build my understanding of why it is so hard to change broken policies (and certainly matches with my “most people I know are well-intentioned and good at their jobs”). When we need something to change, we can’t just blame individuals. We should definitely start thinking of organizations as their connections, not only as their individuals. It doesn’t give a solution yet, though.

Intermission: Computing ponds and rabbit hole¶

This was about the idea of a pond of decaying matter sending signals to each network and becoming sentient or otherwise a network of intelligence. Or other ways of making systems of “dumb” objects sending signals to each other to form a larger whole that appears capable of purposeful action.

As a person in computing, this all flowed quite naturally and makes sense in the concept of neural nets/artificial intelligence.

Part 2: Pathologies of the system¶

(4) How to psychoanalyze a non-human intelligence¶

This starts off with an explanation of why “cybernetics” these days refers to computers: because people got too good at building transistors and computing devices. It took all the attention from the decision making part of the field.

This chapter talks about how systems can be controlled and analyzed. The main consideration is what kind of information controls it, and this gets to the concept of variety of information. Variety, in my own words, somehow relates to the number of states a system can have, or its inherent complexity. It can also refer to the number of possible outside stimuli or control settings a system has. Variety of internal states (or complexity) needs to somehow be aligned with variety of control.

Example: Let’s take a room’s thermostat. It has one input, the temperature, and has one output, amount of heat applied. The variety of control matches the variety of internal states of the system. If we wanted to control temperature and humidity, but only have heating control, it is quite possible that we can’t react to all possible states of the room and bring it to the desired state.

When you have a complex system consisting of lots of black boxes, there are many different states you can be in. What happens if you can’t measure enough information about the state, or have enough control knobs to affect it? You have a problem. The Law of Requisite Variety says you need as much control variety as you have variety of internal states.

This is a general argument and note that it doesn’t require knowing exactly what the measurements and controls are. You can reason that a system has lots of internal variety but not that many ways of control, and know you have a problem - even without knowing exactly what the states or controls are.

This chapter resonated with me and I like the general theory of variety. It provides me a powerful tool to say “you are over-simplifying the policies you are making” - though I imagine it will be hard to get the right people to understand this, if they don’t want to understand what is wrong with their policy from the policy itself.

(5) Cybernetics without diagrams¶

This tries to get into the deep theory of cybernetics without being too technical. It says this is hard, and perhaps it’s not so interesting. However, one part was quite interesting:

A system that maintains its state needs different parts to be a viable system that is capable of maintaining control over itself. There are different standard functions needed to stay in homeostasis with its environment. The cyberneticists thought that such a system needs to have:

System 1: operations. What the system actually tries to do for the real world. It does the main thing (say, run a computing cluster), but left alone it could go out of control and over- or under-build what is needed or not coordinate maintenance and use.
System 2: regulatory. This is the basic regulation of operations to make sure that operations stays synced. In our computer cluster example, it may coordinate hardware upgrades, software installations, and maintenance to make sure the system works. This makes the basic resource bargains for how stuff works. This is about preventing clashes and managing conflicts (things everyone thinks are necessary).
System 3: optimization or integration. This directs different components towards a main purpose and allocates things between different System 1 operations. This is about the overall mission (compared to System 2). This is the first real “management” level and what some people complain about being unnecessary. In our cluster example, this might include allocating resources from hardware upgrades to support and vice versa depending on overall needs of the users.

Systems 1 and 2 absorb most of the variety of customers, but sometimes customers will need something beyond what they can provide. In this case, System 3 needs to step in and adjust the mission by escalating to the level above. Too much System 3 involvement in System 1 (rules with no exceptions) can result in breakdown, since System 3 can’t process all the variety coming in to System 1. This is micromanagement and takes time away from what System 3 should be doing. The other system, of leaving everything to System 1, can result in no planning. The book makes a point that there can be an informal System 3 by System 1 managers; this is OK as long as they make sure it gets done - this is the value of informal meetings.

System 4: intelligence. If System 3 manages what is happening now, System 4 is planning for the future by looking outwards to the world (not just the current state but predicting what is coming). In our cluster example, this might be looking at emerging technologies that change the way clusters are run.
System 5: philosophy or identity. In some sense this balances changes identified by System 4 with the need to keep unit running from System 3 (you can’t change too fast or you lose your identity, and then maybe you cease to exist). It balances the broad organizational units. System 5 decides what the system does, and thus its purpose. In our cluster example, this might involve thinking of the role of clusters as cloud computing and AI become more common.

These systems are recursive throughout the organization. The whole organization has its five systems, then each sub-unit will have its five systems, and so on all the way down.

It doesn’t say these have to be different people or units, just that they need to happen. I do sort of wonder how rigorous the justification is for separating all of them (the book didn’t go into detail about how they were decided).

These systems also have their own amount of variety of signals going in and out. They variety in control signals to each system to be matched to the complexity of the task they are performing.

If they don’t have enough variety to handle an emergency, they need to be able to “declare an emergency” and call a higher level in order to bring the resources needed to solve a problem. The book uses the example of a red handle in a train that means you have to do an emergency stop and notify the control center. All schedules are now messed up and you need a higher level of the organization to fix the problem. (This doesn’t just mean a higher System number; it means the higher recursive “viable system” level with all its boxes).

While I wouldn’t go building these into my teams, it is a good checklist I would use to make sure our team is handling everything that needs to be done. It also provides some good abstraction levels for dividing up tasks, once a team gets large enough.

Intermission: decerebrate cats¶

This tells a metaphor about how you can cut the link in a brain between the cerebellum and the rest of the brain. The living being still seems alive (walking, eating, grooming itself, etc.). But by any measure, it doesn’t think or have purposeful actions anymore. It can keep living and functioning as long as the environment is what it is used to, but once the environment changes, it can’t adapt. This was apparently part of real experiments done.

Stafford Beer talked about decerebrate organizations which similarly have lost the ability to adjust to the world. It will keep surviving until a shock comes up, and then goes into crisis. If it can’t re-establish the higher-level Systems, it will simply collapse. It warns that if someone sees their organization decerebrate-ing itself, start updating your CV and looking for new jobs.

The book gives the example of universities as typical decerebrate organizations, and this is when I started getting sad. It’s exactly what I have been seeing with some parts of my university, in particular about challenges challenges with legal compliance processes or data classification. The world is more complex than it was before and it’s trying to apply old processes to new things, but no one can take a step back and say “wait, we need to adjust how we see ourselves to fit the current state of the world.” It might be more about feedback rather than the System 5, but still I got feelings from reading this

(6) Economics and how it got that way¶

First, what is an economist? The book says that economics is the science of optimizing scarce resources (and since everything is scarce, economists get to play the rulers of society by making models to say the best way to run everything in the world). It does this by trying to express everything in terms of money.

Like many fields, economists have “physics-envy” and have tried to make models and predictions. However, economists can often forget that these are just models and many assumptions have gone into them, and the assumptions can pre-determine the conclusions. You can also forget that the model is just a model, and the real world can do things outside what you predict (this is especially the case since humans are complex) - it calls this the Ricardian Vice.

This is obvious to physicists but worth making clear: just because something is proven in your nice model with the complexity stripped out doesn’t mean it is true in the real world.

One of the dominant models is that an economy is so complex it’s impossible for any normal control methods to handle it - you can’t bring enough variety to control all the components. Thus, the assumption is that a free market composed of many different parts making their own contracts and looking after their own needs serves a computing fabric that can work to control the economy.

Everything can combine into a “the market knows best” (if you make the right assumptions), with the market serving as a computing fabric for making complex decisions - and is the seed of unaccountability.

“Market as computing fabric” matches with mainstream western thought, and I don’t really object to that. Later the book explains what is wrong beyond “capitalism bad”.

(7) If you’re so rich, why aren’t you smart?¶

This starts by talking about how economics is a theoretical and not practical science. Economics is not accounting and most economists couldn’t deal with real businesses, and the book claims this is because economists want to be seen as their own special profession above accountants, managers, and similar.

Accounts and accounting are the basic ways to measure a business, and they serve two purposes. One is to control the business, another is to tell outsiders how the business is doing. Accounting is difficult in any big enough business, and while there are standard practices, there are some choices made. The same set of numbers aren’t necessarily good for all purposes (mandatory reporting for public companies vs internal bookkeeping for how it’s going).

Financial reporting also by its nature reduces variety, and makes fewer signals than there are internal states of the company.

So, what signals does one use to manage a company? It’s easy to hire management consultants and they work to optimize the short-term public financial results at the cost of long-term operational value.

We need to be careful that our reports match with what we need to be measuring.

Intermission: Meanwhile, in Chile¶

Stafford Beer went to Chile during its socialist era with the idea of applying management cybernetics to a command economy that was nationalizing industries (and thus had to manage them). They had computers and were trying to get enough information all in one place (the futuristic-looking “operations room”) to manage it. While it sounded like an interesting idea, applying the law of requisite variety makes one think even this wasn’t enough.

In the end, the US supported a coup (as usual) before it could go far enough to get results (or not).

Part 4: What happens next?¶

Now we get fully into the economics and politics of the modern world.

(8) Enter Friedman¶

This section first taught me about Milton Friedman, a name I had heard often but didn’t know the story of. Intentionally or not, he popularized the “neoliberalism” chain of thought which, more or less, said that a corporation can’t know what is best for society, so the best it can do is try to maximize its profits and let the market (the “computing fabric”) work out the rest.

There’s a distinction between the managerial class (those working to manage the companies, they may care about it) and the investing class (or capitalist class) which only cares about profits. The basic idea of neoliberalism was that, if the management class tries to do something for the benefit of society, it’s spending someone else’s money and lacks accountability (taking on the role of a government).

This whole “maximizing shareholder value” concept (combined with “legal personhood” and “limited liability”) are purely artificial. It’s simply a choice that could have been made another way, or not at all. And we’ll see this invented philosophy causes many downstream effects.

The “leveraged buyout” is when private equity buys a company that is under-performing to replace the management or otherwise force it to be more efficient. This was seen as the ultimate threat for any managers that didn’t sufficiently try to maximize their companies’ profits, to “keep companies in line”, so to say. Once bought-out, the companies were given their own debts and forced to cut back anything else in order to repay those debts, on threat of going bankrupt. All together, this replaces any signals of “care about society” with “ignore everything else and maximize your profits in the short term”, which in the end leads to everything falling apart long-term.

This chapter is the thesis of the book, what is wrong with “late-stage capitalism” and all that. In summary, the financial industry is so focused on optimizing profits (one signal) that everything else is falling apart. This matches what I have seen.

(9) The morbid symptoms¶

This starts off summarizing the book (this is my paraphrasing):

The basic problem is that systems/companies need to be able to re-organize themselves as the world changes. However, due to the financialization of everything, all the other signals needed for society/companies to react to the world can’t be processed, because “make profits” and “pay off debt” override it. This leads to society’s polycrisis.

Stafford Beer’s philosophy was to pay attention to information crossing boundaries. As companies get sliced up, managers’ ranks thinned, and functions outsourced, you lose the capacity to convey this information.

Some theorists had, about 100 years ago, thought the function of the investing class was to take risks and provide what the working class needed, while insulating the working class from the ups and downs of the business cycle. Now that is reversed, where people think the system is the investing class works to maximize its profits, using the working class in the process. The rise of things like populism is the “red handle” that things are not going OK.

The chapter ends by saying this is the nature of the current polycrisis. The system is growing and getting more complex. It has been assumed that the free market will take care of things, but the signals of the computing fabric has been subverted and now all it can care about is short-term profit maximization. All the middle layers of communication are slowly being gotten rid of because it’s unnecessary bloat, but really they were the ones carrying the messages. No one protests “red tape (of bureaucracy) holds us together”.

**I agreed with the what is here, but then again I don’t have the knowledge to know if I should agree or not. Still, roughly makes sense from what else I have read about problems with society.**qq

(10) What is to be done?¶

“If you can hear the problem, you can hear the solution.”

As for the problem of private equity and leveraged buy-outs, one proposed solution is that limited liability should be moderated. Limited liability serves as an accountability sink to whoever buys a company. The author thinks this might be enough.

Broadly, what the author thinks is needed is more viability and less maximization of single goals. Right now, financialization has made one goal take full precedence over everything else, and the very limited variety of these signals means so much gets left out of the decision-making process, at least at the highest levels.

Going back to the artificial intelligence metaphor, we have made an artificial intelligence, told it to maximize profit, and it’s going wild. Every system set up as a maximizer needs another system watching over it.

The book hypothesizes that things like social media provides more ways for more signals to be sent, but how will all these be managed? Can they be? Will AI (another maximizing system) be assigned to do it? It’s just another system that can be blamed. I read this as idle thoughts rather than a fleshed-out proposal (and personally I would be quite hesitant to go down this path, but it’s an interesting thought).

Overall I didn’t quite think there was a great conclusion for the book, but as the first quote says, maybe that’s not the point. This is hard, and if we can see the problem, maybe we can work towards it. I’m hesitant to for any solution that involves social media sending signals, but it is an interesting thought to come back to. Anyway, I don’t need a conclusion or recipe for what to do about our economy, I’m happy with the lessons I learned along the way.

Previous: News channels in chat