Blog - Posts by Richard Darst

Book review: The Unaccountability Machine

2026-09-13T00:00:00+00:00

I’ve been interested in why organizations work (or seem to not work, despite everyone in them being good people). Another book cited the book “The Unaccountability machine” (wikipedia, HelMet libraries) as being related to this topic, so I checked it out. In the end, the book did provide some thoughts on this, mainly through the lens of complex webs of decision making and accountability sinks. The first half was most relevant to me; the second half ventured into economics and society (which was still interesting, but not my main focus).

Summary

In modern times, the word “cybernetics” makes one think of human/machine combinations or maybe computer technology in general, but the original definition (1940s and onward) was about decision making, feedback, and control of systems (machines, organizations, economies, etc.). It’s important to keep this in mind, otherwise the topic and review below can easily get confused with unrelated technology matters. Cybernetics and the book are about information flows.

Systems don’t have motivations. They are made up of people who have motivations and outside influences. How can a system made of good people have bad outcomes? How can it be avoided? These are the kinds of questions I wanted to answer.

There are several main lessons I got from the book:

The number of states a system can be in vs the amount possibly gradations of control input. This is known as “variety”. An example from my university:

The university defines there are four categories of data (levels of confidentiality), and that should dictate how the data is handled. However, a university is a complex thing with very different {operations, teaching, research} divisions, and four categories to control data (input variety) just isn’t enough to express the wide variety of actual data (and there isn’t enough variety of other processes or exceptions). This leads to people doing whatever they want (ignoring the rules since they don’t seem applicable), or never-ending discussions about how the system needs to be tweaked to make it work.

The idea that the internal complexity of a system has to match the external complexity it faces is the Law of Requisite Complexity, and once I realized this, I started seeing almost every dysfunctional thing of our university being caused by its violation.
Cyberneticists have broken down organizations into five systems needed for their control (Viable system model) (operations, regulation, optimization or integration, intelligence, and philosophy or identity, see more in Chapter 5’s summary below). My own team is too small to have these as defined roles, but thinking of them helps me realize some things we need to make sure get done.
Any complex organization isn’t one mind, or many individuals, but many different parts (people) communicating in a complex network. This can easily allow decisions to be made and propagated elsewhere in a way that doesn’t quite make sense, with little feedback or accountability to the place the decisions were made. This makes the accountability sinks (which was apparently defined here, though I would have thought it was older).

A complex network of nodes sending signals to each other, sometimes having a result that is undesired? Sounds like “artificial intelligence” these days, with hallucinations and wrong answers, doesn’t it? Just like with “AI”, there needs to be a feedback system to keep an eye on things.

The second half of the book is focused on economics and society (because the author is somewhat of an economist) and less relevant to my work, but I found that it was quite useful to me anyway. Overall, I found some good points that I could apply to my own non-economics life.

Overall thoughts

I liked the book and was happy to read something that combined something in my comfort zone (science-ish description of cybernetics) with something outside it (economics). I’ve learned some buzzwords that I can use in my job to push for changes, though I predict it’s an uphill battle to get anything actually done.

As my team gets more complex, we need to encapsulate different parts and have their internal variety and management, while having a more limited variety that reaches higher levels for the systematic decision making.

Chapter-by-chapter analysis

Below is a summary of each chapter. I state what the book says (with some of my opinions or take-away messages mixed in), but I am not qualified to judge if the book is correct or not.

Part 1: The nature of the crisis

(0) Introduction

The industrialization of society makes larger and more complex organizations, and they need to be controlled. The theory of these control structures is “cybernetics” (the original definition).

I liked this intro, because it related to my thing about industrialization (of finance, control, etc) being the more relevant problem in our society than just capitalism that most people complain about. The idea is that when things get too big, they somehow makes more than their share of problems.

(1) Something’s up

Organizations sometimes do things that just make no sense, for example, the banking crisis of 2008. No person wanted this, yet somehow it happened. There were even people who could predict it, but no one would listen to them because the system was on autopilot and not taking feedback.

Accountability sinks are ways that decisions can be made such that no individual person is accountable. A decision may come out of the organization, but all the individuals of the organization can say “I don’t agree with this but it’s not me who decided” - so where does it come from? Who takes accountability? These are quite common when you start looking for them, when everyone can pass off the responsibility for a decision/rule to someone else (often laundering it through various policies or laws).

I will choose to not mention one example the book gives involving animals, since it’s just too tragic. Another example given is academic publishing. Professional publishers have managed to insert themselves into the academic system and extract huge amounts of money, and most people agree there are serious problems with relying almost exclusively on citations for academic career purposes. Yet, still, it stays, because it’s now part of the system and the politics of making any other decision about careers would be too much.

There are two types of decision-makers, kings (who are humans and to whom decisions can be appealed), and priests (who speak the word of god and thus you can’t appeal to them). Too often, a decision may be made by a “king” but gets conveyed by “priests”, and there is no way to send feedback that something is wrong. This can cause unrest.

All the above isn’t just an accident. It’s a trade-off, discretion (to do the right thing) vs following the rules. What’s the balance? This is the next chapter.

Overall, this chapter said what I wanted to hear about organizations, bad choices, and and unaccountability and left me wanting an analysis.

(2) Stafford Beer

This chapter is about the historical context.

Stafford Beer was one of the founders of management cybernetics - as discussed above, how decisions get made. Beer looked at systems as composed of black boxes (something with defined inputs and outputs, but the internals can’t be examined): different components, and each component had some behavior. You couldn’t/didn’t look into the boxes. Of course, you could recursively look into them if you wanted, or package multiple black boxes into another black box. Still, at some level, one has to draw black boxes around some parts of the system and analyze the parts outside of that.

Management cyberneticists looked at systems as black boxes and that interact with each other, to discover various theories of control. Thinking in terms of black boxes gives the saying “the purpose of a system is what it does”. This is important to keep in mind so that one doesn’t blame the individuals for what the system does - you have to look at the connections and control structures.

In the end, systems/organizations are controlled by meetings between the different levels and components. This often happens by resource bargains between different components, so that each part can focus on its internals with a defined interface to the outside.

I understand the concept of the black boxes, it took some time to fully internalize the recursive nature of them. I’m still not quite clear where one should decide to draw the black boxes (what level and what size), but I guess that’s part of the art of the field. You should probably draw them at the places that are most useful for your current problem. In the end, this question doesn’t matter too much for my understanding.

(3) Aliens among us

So, a system is made up of black boxes once you go deep enough. These are all connected and sending signals to each other. It becomes a… neural network? There are clear metaphors to “artificial intelligence” here, where the system takes on a mind of its own, but each individual part can’t have accountability. This also relates to the Chinese room (translation) thought experiment.

This applies to all kinds of things, from social media algorithms to corporate management. Yet, we know the organization itself is not sentient. So, the organization itself isn’t accountable, but neither are the independent parts (because they are just playing their role).

The more connections a system has, the harder it is to understand. However hard you look, you will probably miss something.

If you are interacting with one of these systems, you can be given the policies/documents in response to a query about why a decision was made a certain way. But that never explains why the policies are made a certain way. The accountability for them has been sunk into the network itself, and if people only try to hold individuals accountable, everyone can escape accountability.

So, in the end, you see the same thing over and over again. Comments about an organization that say they admire all the people inside, but despise the overall organization.

This part of organizations being a computing network really made sense to me and started to build my understanding of why it is so hard to change broken policies (and certainly matches with my “most people I know are well-intentioned and good at their jobs”). When we need something to change, we can’t just blame individuals. We should definitely start thinking of organizations as their connections, not only as their individuals. It doesn’t give a solution yet, though.

Intermission: Computing ponds and rabbit hole

This was about the idea of a pond of decaying matter sending signals to each network and becoming sentient or otherwise a network of intelligence. Or other ways of making systems of “dumb” objects sending signals to each other to form a larger whole that appears capable of purposeful action.

As a person in computing, this all flowed quite naturally and makes sense in the concept of neural nets/artificial intelligence.

Part 2: Pathologies of the system

(4) How to psychoanalyze a non-human intelligence

This starts off with an explanation of why “cybernetics” these days refers to computers: because people got too good at building transistors and computing devices. It took all the attention from the decision making part of the field.

This chapter talks about how systems can be controlled and analyzed. The main consideration is what kind of information controls it, and this gets to the concept of variety of information. Variety, in my own words, somehow relates to the number of states a system can have, or its inherent complexity. It can also refer to the number of possible outside stimuli or control settings a system has. Variety of internal states (or complexity) needs to somehow be aligned with variety of control.

Example: Let’s take a room’s thermostat. It has one input, the temperature, and has one output, amount of heat applied. The variety of control matches the variety of internal states of the system. If we wanted to control temperature and humidity, but only have heating control, it is quite possible that we can’t react to all possible states of the room and bring it to the desired state.

When you have a complex system consisting of lots of black boxes, there are many different states you can be in. What happens if you can’t measure enough information about the state, or have enough control knobs to affect it? You have a problem. The Law of Requisite Variety says you need as much control variety as you have variety of internal states.

This is a general argument and note that it doesn’t require knowing exactly what the measurements and controls are. You can reason that a system has lots of internal variety but not that many ways of control, and know you have a problem - even without knowing exactly what the states or controls are.

This chapter resonated with me and I like the general theory of variety. It provides me a powerful tool to say “you are over-simplifying the policies you are making” - though I imagine it will be hard to get the right people to understand this, if they don’t want to understand what is wrong with their policy from the policy itself.

(5) Cybernetics without diagrams

This tries to get into the deep theory of cybernetics without being too technical. It says this is hard, and perhaps it’s not so interesting. However, one part was quite interesting:

A system that maintains its state needs different parts to be a viable system that is capable of maintaining control over itself. There are different standard functions needed to stay in homeostasis with its environment. The cyberneticists thought that such a system needs to have:

System 1: operations. What the system actually tries to do for the real world. It does the main thing (say, run a computing cluster), but left alone it could go out of control and over- or under-build what is needed or not coordinate maintenance and use.
System 2: regulatory. This is the basic regulation of operations to make sure that operations stays synced. In our computer cluster example, it may coordinate hardware upgrades, software installations, and maintenance to make sure the system works. This makes the basic resource bargains for how stuff works. This is about preventing clashes and managing conflicts (things everyone thinks are necessary).
System 3: optimization or integration. This directs different components towards a main purpose and allocates things between different System 1 operations. This is about the overall mission (compared to System 2). This is the first real “management” level and what some people complain about being unnecessary. In our cluster example, this might include allocating resources from hardware upgrades to support and vice versa depending on overall needs of the users.

Systems 1 and 2 absorb most of the variety of customers, but sometimes customers will need something beyond what they can provide. In this case, System 3 needs to step in and adjust the mission by escalating to the level above. Too much System 3 involvement in System 1 (rules with no exceptions) can result in breakdown, since System 3 can’t process all the variety coming in to System 1. This is micromanagement and takes time away from what System 3 should be doing. The other system, of leaving everything to System 1, can result in no planning. The book makes a point that there can be an informal System 3 by System 1 managers; this is OK as long as they make sure it gets done - this is the value of informal meetings.

System 4: intelligence. If System 3 manages what is happening now, System 4 is planning for the future by looking outwards to the world (not just the current state but predicting what is coming). In our cluster example, this might be looking at emerging technologies that change the way clusters are run.
System 5: philosophy or identity. In some sense this balances changes identified by System 4 with the need to keep unit running from System 3 (you can’t change too fast or you lose your identity, and then maybe you cease to exist). It balances the broad organizational units. System 5 decides what the system does, and thus its purpose. In our cluster example, this might involve thinking of the role of clusters as cloud computing and AI become more common.

These systems are recursive throughout the organization. The whole organization has its five systems, then each sub-unit will have its five systems, and so on all the way down.

It doesn’t say these have to be different people or units, just that they need to happen. I do sort of wonder how rigorous the justification is for separating all of them (the book didn’t go into detail about how they were decided).

These systems also have their own amount of variety of signals going in and out. They variety in control signals to each system to be matched to the complexity of the task they are performing.

If they don’t have enough variety to handle an emergency, they need to be able to “declare an emergency” and call a higher level in order to bring the resources needed to solve a problem. The book uses the example of a red handle in a train that means you have to do an emergency stop and notify the control center. All schedules are now messed up and you need a higher level of the organization to fix the problem. (This doesn’t just mean a higher System number; it means the higher recursive “viable system” level with all its boxes).

While I wouldn’t go building these into my teams, it is a good checklist I would use to make sure our team is handling everything that needs to be done. It also provides some good abstraction levels for dividing up tasks, once a team gets large enough.

Intermission: decerebrate cats

This tells a metaphor about how you can cut the link in a brain between the cerebellum and the rest of the brain. The living being still seems alive (walking, eating, grooming itself, etc.). But by any measure, it doesn’t think or have purposeful actions anymore. It can keep living and functioning as long as the environment is what it is used to, but once the environment changes, it can’t adapt. This was apparently part of real experiments done.

Stafford Beer talked about decerebrate organizations which similarly have lost the ability to adjust to the world. It will keep surviving until a shock comes up, and then goes into crisis. If it can’t re-establish the higher-level Systems, it will simply collapse. It warns that if someone sees their organization decerebrate-ing itself, start updating your CV and looking for new jobs.

The book gives the example of universities as typical decerebrate organizations, and this is when I started getting sad. It’s exactly what I have been seeing with some parts of my university, in particular about challenges challenges with legal compliance processes or data classification. The world is more complex than it was before and it’s trying to apply old processes to new things, but no one can take a step back and say “wait, we need to adjust how we see ourselves to fit the current state of the world.” It might be more about feedback rather than the System 5, but still I got feelings from reading this

(6) Economics and how it got that way

First, what is an economist? The book says that economics is the science of optimizing scarce resources (and since everything is scarce, economists get to play the rulers of society by making models to say the best way to run everything in the world). It does this by trying to express everything in terms of money.

Like many fields, economists have “physics-envy” and have tried to make models and predictions. However, economists can often forget that these are just models and many assumptions have gone into them, and the assumptions can pre-determine the conclusions. You can also forget that the model is just a model, and the real world can do things outside what you predict (this is especially the case since humans are complex) - it calls this the Ricardian Vice.

This is obvious to physicists but worth making clear: just because something is proven in your nice model with the complexity stripped out doesn’t mean it is true in the real world.

One of the dominant models is that an economy is so complex it’s impossible for any normal control methods to handle it - you can’t bring enough variety to control all the components. Thus, the assumption is that a free market composed of many different parts making their own contracts and looking after their own needs serves a computing fabric that can work to control the economy.

Everything can combine into a “the market knows best” (if you make the right assumptions), with the market serving as a computing fabric for making complex decisions - and is the seed of unaccountability.

“Market as computing fabric” matches with mainstream western thought, and I don’t really object to that. Later the book explains what is wrong beyond “capitalism bad”.

(7) If you’re so rich, why aren’t you smart?

This starts by talking about how economics is a theoretical and not practical science. Economics is not accounting and most economists couldn’t deal with real businesses, and the book claims this is because economists want to be seen as their own special profession above accountants, managers, and similar.

Accounts and accounting are the basic ways to measure a business, and they serve two purposes. One is to control the business, another is to tell outsiders how the business is doing. Accounting is difficult in any big enough business, and while there are standard practices, there are some choices made. The same set of numbers aren’t necessarily good for all purposes (mandatory reporting for public companies vs internal bookkeeping for how it’s going).

Financial reporting also by its nature reduces variety, and makes fewer signals than there are internal states of the company.

So, what signals does one use to manage a company? It’s easy to hire management consultants and they work to optimize the short-term public financial results at the cost of long-term operational value.

We need to be careful that our reports match with what we need to be measuring.

Intermission: Meanwhile, in Chile

Stafford Beer went to Chile during its socialist era with the idea of applying management cybernetics to a command economy that was nationalizing industries (and thus had to manage them). They had computers and were trying to get enough information all in one place (the futuristic-looking “operations room”) to manage it. While it sounded like an interesting idea, applying the law of requisite variety makes one think even this wasn’t enough.

In the end, the US supported a coup (as usual) before it could go far enough to get results (or not).

Part 4: What happens next?

Now we get fully into the economics and politics of the modern world.

(8) Enter Friedman

This section first taught me about Milton Friedman, a name I had heard often but didn’t know the story of. Intentionally or not, he popularized the “neoliberalism” chain of thought which, more or less, said that a corporation can’t know what is best for society, so the best it can do is try to maximize its profits and let the market (the “computing fabric”) work out the rest.

There’s a distinction between the managerial class (those working to manage the companies, they may care about it) and the investing class (or capitalist class) which only cares about profits. The basic idea of neoliberalism was that, if the management class tries to do something for the benefit of society, it’s spending someone else’s money and lacks accountability (taking on the role of a government).

This whole “maximizing shareholder value” concept (combined with “legal personhood” and “limited liability”) are purely artificial. It’s simply a choice that could have been made another way, or not at all. And we’ll see this invented philosophy causes many downstream effects.

The “leveraged buyout” is when private equity buys a company that is under-performing to replace the management or otherwise force it to be more efficient. This was seen as the ultimate threat for any managers that didn’t sufficiently try to maximize their companies’ profits, to “keep companies in line”, so to say. Once bought-out, the companies were given their own debts and forced to cut back anything else in order to repay those debts, on threat of going bankrupt. All together, this replaces any signals of “care about society” with “ignore everything else and maximize your profits in the short term”, which in the end leads to everything falling apart long-term.

This chapter is the thesis of the book, what is wrong with “late-stage capitalism” and all that. In summary, the financial industry is so focused on optimizing profits (one signal) that everything else is falling apart. This matches what I have seen.

(9) The morbid symptoms

This starts off summarizing the book (this is my paraphrasing):

The basic problem is that systems/companies need to be able to re-organize themselves as the world changes. However, due to the financialization of everything, all the other signals needed for society/companies to react to the world can’t be processed, because “make profits” and “pay off debt” override it. This leads to society’s polycrisis.

Stafford Beer’s philosophy was to pay attention to information crossing boundaries. As companies get sliced up, managers’ ranks thinned, and functions outsourced, you lose the capacity to convey this information.

Some theorists had, about 100 years ago, thought the function of the investing class was to take risks and provide what the working class needed, while insulating the working class from the ups and downs of the business cycle. Now that is reversed, where people think the system is the investing class works to maximize its profits, using the working class in the process. The rise of things like populism is the “red handle” that things are not going OK.

The chapter ends by saying this is the nature of the current polycrisis. The system is growing and getting more complex. It has been assumed that the free market will take care of things, but the signals of the computing fabric has been subverted and now all it can care about is short-term profit maximization. All the middle layers of communication are slowly being gotten rid of because it’s unnecessary bloat, but really they were the ones carrying the messages. No one protests “red tape (of bureaucracy) holds us together”.

**I agreed with the what is here, but then again I don’t have the knowledge to know if I should agree or not. Still, roughly makes sense from what else I have read about problems with society.**qq

(10) What is to be done?

“If you can hear the problem, you can hear the solution.”

As for the problem of private equity and leveraged buy-outs, one proposed solution is that limited liability should be moderated. Limited liability serves as an accountability sink to whoever buys a company. The author thinks this might be enough.

Broadly, what the author thinks is needed is more viability and less maximization of single goals. Right now, financialization has made one goal take full precedence over everything else, and the very limited variety of these signals means so much gets left out of the decision-making process, at least at the highest levels.

Going back to the artificial intelligence metaphor, we have made an artificial intelligence, told it to maximize profit, and it’s going wild. Every system set up as a maximizer needs another system watching over it.

The book hypothesizes that things like social media provides more ways for more signals to be sent, but how will all these be managed? Can they be? Will AI (another maximizing system) be assigned to do it? It’s just another system that can be blamed. I read this as idle thoughts rather than a fleshed-out proposal (and personally I would be quite hesitant to go down this path, but it’s an interesting thought).

Overall I didn’t quite think there was a great conclusion for the book, but as the first quote says, maybe that’s not the point. This is hard, and if we can see the problem, maybe we can work towards it. I’m hesitant to for any solution that involves social media sending signals, but it is an interesting thought to come back to. Anyway, I don’t need a conclusion or recipe for what to do about our economy, I’m happy with the lessons I learned along the way.

News channels in chat

2026-07-31T00:00:00+00:00

I hope we had a low threshold for sharing anything in chat, as long as the sharer puts a bit of effort into a summary. In the CodeRefinery chat, we’ve thought about how to build more interaction in our chat (even though, compared to some orgs, we think we have quite a bit. Perhaps that’s the power of Zulip). My proposal is “news channels”. The goal is like a blog or news aggregator, with the similar goal of educating ourselves and getting to know each other better.

I like seeing a wide variety of links, but time is limited. Not everyone can read every link posted, so I hope that people put their human brain into a summary (suitable for the readers) that is educational even if read alone. Not clickbait, and not “read the link for more”.

The basic idea is that we people want to blog more, but that’s a lot of work. It’s lower threshold to declare some chat channels to be “blog-like” and encourage even off-topic things to go there. Hopefully it leads to more blogging, too.

For example, I propose the CodeRefinery chat declare these as news channels:

#general: News channel for anything remotely related to scientific computing.
#random: News channel for anything else.

“News channel” does not mean that channel is for only news posts, or for things that could be blog post. It means “for things you might want to tell others about”.

This post is my personal opinion, but will be updated with the practices of the organizations I work with (CodeRefinery, Aalto Scientific Computing). You are welcome to copy and modify this. This should not be used to tell people they are doing something wrong, but only to encourage new interaction. You don’t have to do what is here.

What is clickbait?

Philosophy: Clickbait (according to wikipedia) “is designed to attract attention and to entice users to follow (“click”) that link” [note: normal clickbait is often misleading or sensationalized, and that’s not a problem in our chats.]. My main consideration is you need to click through to know the actual story.

To me, anti-clickbait means putting the main message as early as possible so that one can stop reading as early as they want and still get something:

Read the first sentence, get a 1-sentence understanding of the topic.
Read the first paragraph, get a 1-paragraph understanding of the topic.
Read the first section, get a 1-section understanding of the topic.
Apply this recursively at every level.

This requires more thought in writing, but is a service for the reader. I think we should strive more towards this [note: Does this post meet these criteria? I usually try to, but I don’t always manage.]. It’s related to Bottom-line Up Front communication <https://en.wikipedia.org/wiki/BLUF_(communication)>__, from which I got some of these ideas.

Basics

This is my proposal:

News channels are non-work channels, and you don’t have to be subscribed. Don’t expect all your colleagues to be subscribed. If you have an announcement for a team, make sure you mention it in a work channel, too.
You can post anything within the scope, for example:
- Link to other content you have found interesting.
- Link to you blog posts or things you have created.
- Pictures you have taken.
- Original ideas that might become blog posts later.
- Things you have learned (doesn’t have to have a link), watched, etc.
- A question or thought you have (including something you might blog about later).
Put some human effort into describing it (see below). Please no bare or low-context links. Yes, people can read things themselves, but there’s too much. Our networks helps filter summaries.
There’s not a limit (yet) on the amount you can post, but in general, the more you post, the more human effort you should put into describing things. People can independently subscribe to wherever you get material from if they want everything from there.
Follow-up discussion on any content is encouraged, but not required. Don’t get discouraged if no one reacts. It might mean your description is good enough. Do react to interesting things to encourage more of that type of thing.

You are especially encouraged to post anything new you create, no matter how small it is. This is how we learn of each other’s skills.

How-to

Further thoughts, adapt as is appropriate:

Create a new topic for each new thing. Don’t worry about making too many topics, but you can always reply with more stuff for relevant old topics if it’s relevant. Topics are the way someone quickly filters through it, and remember they will “Mark all as read” once they have browsed enough. People may look through their Zulipchat inbox at topics and decide what to click or not, so make it easy for someone to know if it’s relevant to their interests.
You can use a detailed topic description, for example:
- “TIL: uv for python package installation”
- “OpenAI has acquired Astral (the company behind uv)”
- “venv-apptainer: new script for containing virtual environments”
Include a description - not just a bare link or wall of text. You should put some human effort into the description, even if only one sentence. If you can do this, we will be happy with whatever it is. The description should be enough for someone to learn some bare basics if they don’t read further (and know if they should read more).
- At minimum, an initial description that stands alone and provides some education. Not just clickbait, but a real description. Can be anywhere from a sentence or two, to a paragraph. People have a limited time and attention, so write for that.
- You are welcome to go as far in depth as you would like after that… long info is OK.
- Especially consider giving your own thoughts on the matter, if it’s by someone else. Do you agree with it? Disagree? Have further ideas? These kind of reviews make links far more interesting, because many people don’t have the full context to understand things.
Even if you wrote something elsewhere and you are linking to it, give a summary in the chat.

A reminder: Do not use this blog post to tell people they are doing something wrong. Don’t think the barrier is too high. Don’t assume this post is the best practices. We want you to do your best, and read what others do, and everyone to continually improve.

The benefits?

Does this seem like a lot of work? Well, yeah, but it’s the trade-off to engagement. In the age of generated content, we want human effort. Look at it this way: thinking about, writing, and reading these summaries is good general communication practice with people outside your field.

“Code” vs “wiki” (vs “book”) rules for project contributions

2026-07-20T00:00:00+00:00

I have sometimes said the phrases “wiki rules”. or “cade rules” about contributing to a project. What does this mean? Roughly, “code rules” is being careful and making sure everything is correct before finalizing a contribution. Wiki rules is just doing something and letting it get improved later. Both ways have their place, and I think sometimes we can get so obsessed with being “right” that we forget to get stuff done. I’ll discuss below.

Code rules

This comes from contributing to code projects, where there are often standards for a review before merging anything, exhaustive tests to make sure that everything new is correct, and there should be no possibility of regression. The basic philosophy is that an error anywhere can break things everywhere, so try to prevent errors from making it into the project.

For stable, mature projects , this makes sense. It can extend to non-code things, too, when the expectation is “do nothing wrong”: say, you are organizing a big event and you need to have a consistent message to attendees for everything you publish.

Code rules requires enough time to review and fix things up. It could be done without contribution guidelines, but then you spend more time reviewing and it can have a demotivation problem, if the review is perceived as too harsh. With extensive contribution guidelines, the barrier to contribution becomes higher.

Examples of this include:

The general idea that one should make a pull request, get it reviewed, and if it’s OK, then merge it.
CodeRefinery would sometimes pedantically teach that all code should be reviewed, even if you are the main person of a project - since it improves quality and teaches others what is going on.
Look at most mature project’s contribution guidelines and they are usually somewhat specific and include reviews. At a big scale, this is needed.
- Numpy contribution guidelines
- Extensive contribution guidelines for the book “The Turing Way”

Wiki rules

On the other hand, wikis usually operate on the idea of “press ahead”, make your change directly, and if so anything isn’t perfect, it will be iteratively improved later. If people disagree, there are ways to resolve conflicts later on. There are few, if any, checks before committing to the history. This doesn’t mean there aren’t standards and expectations, but there is tolerance if new contributors don’t do it just right in their contributions.

For a new, rapidly changing project with few contributors, this can make sense. If there are few contributors who communicate a lot, it is easy. If there are many contributors, there need to be good standards (as there is some hope of convergence) and enough contributors to converge to those standards. Wiki rules requires a strong mental model that everyone shares.

Disadvantages include the time needed to fix things up (or even notice the wrong thing…). It can also be demotivating if someone goes and makes a big change, and it gets iterated to significantly change it. (Sound similar to code rules, right?)

Examples of this include:

Wikipedia’s “Be Bold” guide page
[According to my memory,] The OpenStreetMap wiki (contributor instructions) used to have a page “Press Ahead” with a message to just add things to the best of your ability and it will be improved later. I can’t currently find the page, I presume because there’s enough maturity that stability is more important now.

New one: book rules

Added 2026-09-14: There is one more possibility, “book rules”, where there is an author who is taking overall editorial control, and there are usually not direct changes by others. There may be suggestions, by others, but usually one person listens to all the suggestions, decides the direction, and chooses the final wording. Any review may end up essentially re-writing it to get the exact phrasing the main author wants.

In this, contributions are of the form of ideas and suggestions without exact wording being provided.

Examples of this might include:

Opinion pieces such as my notes for academic supervisors.

Summary

I guess it all comes down to “don’t let the perfect become the enemy of the good”. When there are few people and you communicate often, you can iterate and improve. If you get large and stable (or people depend on complete correctness), then maybe it’s time to switch to code rules.

I think the difference is not exactly “are there strict contribution guidelines?”. The difference is if someone should contribute something possibly wrong and let it be fixed later, or strict pre-review. Every project needs to find their balance of amount of contribution guidelines, amount of prior review, amount of other contribution, and wiki/code rules.

Examples

A project could have an in-person meeting to make a plan, then switch to wiki rules to implement the plan, then a round of review and discussion followed by more iteration. After it has stabilized, moves towards code rules.

Early in planning a conference, we use wiki rules to combine all our ideas onto the website (before anyone even sees the site) and refine them, aggressively narrowing down. To avoid wasted work, it’s “first-draft” or “notes” level work at the beginning if we aren’t sure about it.

Debian with unionfs root

2026-07-13T00:00:00+00:00

I wanted to make a virtual machine that has a root that is immutable - when booted, it would have a tmpfs for all files, and then when reboots, everything goes back to normal. I set this up before the read-only root filesystem, but the use case is slightly different here. I want a way to test things on the internet, without going through my home network, more lightweight than making/snapshotting and deleting a VM every time. It might also be useful when you want a system that is mostly normal but home directories are in memory.

This post is mostly internal notes but may be helpful to others.

The idea

I make a VM and give it an overlayfs for /home. This has the base (the real /home and the upper directory /tmp/home where any modifications are stored. /tmp is an automatic tmpfs, and then there is no permanent changes made to one’s home directory.

First, install the VM and I have found it is good to touch ~/.Xauthority before doing the overlay, so that ssh -X works.

Filesystem mounts

This is implemented via an overlayfs.

On the Debian version I have, /tmp is already a tmpfs. If not, you should set up that first.

I add the following into /etc/fstab. It seems to automatically make the directories within /tmp, which of course are never saved since it is a tmpfs.:

homeoverlay   /home/          overlay  auto,nofail,unbindable,lowerdir=/home/,upperdir=/tmp/home/,workdir=/tmp/home-work 0 0

The following line makes a bind-mount of root which I hoped to use to access the underlay of /home without having to reboot, but this line did not work that way (the binds were passed through, even though it is bind and not rbind which should be recursive). Any other bind-mounts I tried to do did not work as expected:

/             /mnt/root-bind/ none     defaults,bind 0 0

To bind-mount (via qemu or whatever it was) from outside, I have this fstab line:

/new          /new/           virtiofs defaults,auto 0 0

Network bridge to VPN

The network is bridged to a separate vlan to further isolate it (this vlan is managed by my home router outside the host, so there is no way this system could possibly send traffic without the vpn). In virt-manager, I set it to a bridge device I have configured on the host to be bridged to the VPN. I’m recording this here since it’s relevant.

First, you need to make a bridge device. I add the following to /etc/network/interfaces. I don’t make this automatically activate, since on the host I don’t want long-term effects unless I do something. The VM fails to start if it isn’t activated:

iface br-vpn2 inet dhcp
  bridge_ports enp4s0.22
  metric 300

Activate the bridge with ifup br-vpn22.

Modern Linux send bridge traffic through netfiter (iptables). You can either allow it in the FORWARD chain if it’s not set by default, or for my case it was easiest to make it not use netfilter:

# echo "0" | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables

If you don’t do the above, remember to sysctl net.ipv4.ip_forward=1.

Zulipchat guide: managing too much information

2026-06-29T00:00:00+00:00

Zulip is a chat platform (open-source and self-hostable). I think it is very good, but often confusing to newcomers. This post explains how I see it and how I think others can manage all the information. This isn’t meant to be a Zulip basics guide.

I can’t recommend Zulipchat enough if you want a proper online community. I’m not paid to say this.

The fundamental contradiction of chats

I think it’s important to start with why I believe people have a problem with Zulipchat. It’s because it works. People use it to talk, and it leads to thriving chat communities. This leads to too much information. It doesn’t lead to a large number of one-to-one or group direct messages, but functioning communities.

On the other hand, it has good tools to filter things and let you focus on what you want. It takes some getting used to, but it’s possible.

So, don’t be afraid. You are joining a community which has decided to be a community, and that community is probably here to help you with that. The cost is that you need to manage your time, so read on.

Zulip basics

The main concepts are:

Channels (formerly known as “streams”) are the basic organization, like in other chat systems.
Topics are like email subjects of threads within a channel, which have no similar concept in other chat systems. This is what lets someone get a summary of what is going on and manage the flow. You can mention @topic to get just those active in a topic.
Messages are as expected, but all messages have both a channel and topic. (These days you can actually send a message without a topic and it appears as “general chat”, but I would recommend avoiding this as possible.
Your view can be narrowed to certain channels, topics, or other search terms. There are a wide variety of places to click to do this (from almost any view). Click around and see how it works - it will save you lots of time. If you click on a channel/topic name, it narrows to that channel/topic around that message.
If you click on actual messages, it opens reply boxes in that topic.

This has two major implications. First, you have to think a little bit before you start to type. This slows you down by a few seconds, but it’s not so hard and worth it to think “what am I talking about here? Is there an existing thread to join?” But don’t worry too much, a quick search is enough.

Second, once you have things sorted, you can much more easily filter and find what’s important to you. You can see more of past conversations. If you see a new interesting message, you can click on it and see all the past history. You can get notified for specific conversations and hide others.

As a newcomer, you won’t know what all has happened before, so thus may start duplicate topics. That’s OK! In all the organizations I am part of, there are some people who act as “curators” and organize topics, combining or splitting them. They may combine, split, or move yours. This isn’t bad. Just chat and let others deal with that.

Notification levels

There are three levels of following/notifications:

Following where you get notifications (sound/pop-up/email/etc.)
Default
Muted where you don’t even see the messages by default, but if you get @-mentioned then you do. You can then click on the message and see the whole topic.

In your personal settings, there are a variety of settings to make yourself auto-follow anything you interact with (send messages to) or get mentioned in.

Also, every one of these can operate per-channel or per-topic. Thus, you have lots of flexibility. (Of course this leads to the paradox of choice, where more options decrease happiness).

Zulip conventions

My organizations tend to:

Not @-mention people as much as in other places (it can be done when you need to, but if you know someone actually follows the chat, they can usually find the relevant topics). People set the auto-follow depending on their needs.
Not quote-replying that often, unless it’s not very recent in the topic or you want to emphasize it. People can always click the topic and see history, no matter when they see it.
Have fewer channels and rely more on setting topics.
Rely less on private messages.
Do make a search for any past topics and re-post to them if it’s relevant.
Do rearrange messages after they are sent, as topics diverge and converge.
Make new topics frequently, but also split/join topics often when they drift.
“Curators” who have been around a while often move/organize topics that other people start, with a low threshold (this doesn’t mean something went wrong, and it may connect you to older info). But anyone is welcome to do this.

Strategy: if you follow selectively

If you are like most people and can’t follow everything, that’s OK. When you check chat:

Check your “Inbox” view. This shows everything new and unread (and you can limit it to where you have been mentioned/where you are following).
Alternatively, check the “Recent topics” view. This will show you every topic that has been active, sorted by most recent. You can read the topics and see if there’s anything interesting. When you click on a topic, you can scroll back to all the old history, not just the unread messages.
If you want, you can scroll the “Combined feed” that shows everything if you want. This has all the messages mixed together, which will be hard to follow, but you might see something that catches your eye. Click the topic name and you will narrow to that topic and see the whole thing.
Then, and I can’t emphasize this enough, mark everything as read. It’s in the past, you won’t catch up. If something is still relevant, it will re-appear in the recent topics and you can still see all the history.

A variant of this is to go click on each channel separately on the sidebar, which makes its recent topics appear. You can then either browse the channel in chronological order or click on the relevant topics to narrow to them.

You can select “Inbox” or “Recent topics” as your default view.

Even if you try to follow everything, when you come back from a vacation or similar, do this and “mark all as read”.

Strategy: if you follow everything

Some people want to try to follow everything. In this case, set combined feed as your default home view (Personal settings > Preferences) and you will see every message in chronological order (with all the topics interleaved). This lets you quickly follow new things, but can easily become overwhelming. When you see something that is interesting, you click on its topic and see the history of that topic.

You might want to disable the auto-follow settings (in your personal settings). Since you follow everything on your own time, you don’t need to

In reality, I often combine these strategies. I’m one of the topic curators, so I tend to follow everything. Still, if I’m away from a few days it’s too much, so I click through each channel separately to catch up by-topic before going back to the combined feed.

Strategies to follow what’s important to you

For your most important channels, you can pin the to the top.
“Follow” certain topics so that you get notified. I do this for topics such as “lunch” (“where are we eating today?”)
If you don’t look often, change your personal settings to automatically follow any topic you get @-mentioned in or interact in. Un-follow them if/when your input is no longer needed.
Set up alert words (personal settings) which tag certain words as @-mentions.

Strategies to ignore what is not important to you

You can be in a channel but have it muted, so you don’t see it by default, but you can be mentioned within in (and you can see it when you click on it). If there’s a private channel, this is probably a better strategy than not being in it (since you can’t as easily re-join). This also means it’s easy to occasionally click on it to see what’s going on, while not taking up mental space on a daily basis.
Mute specific high-volume topics that are not relevant to you. I do this for some high-volume automated alerts.
Change the “followed messages count” to only include followed topics, DMs, and mentions” - so that indicator isn’t always taking your mental focus.

Emoji reactions

The organizations I am in often use Emoji reactions. This works like other places, but there are some conventions I try to use (note these are something I think, not an official policy or anything):

For expressing agreement/disagreement:

🔼 (:upvote:): Reacting to a concrete suggestion, it means “in no uncertain terms I approve of you taking that action”. I use this, and the following four, since “no one is answering, what should I do?” is a big problem with decisions via chat.
⏫ (:double_up:): I agree with you and am willing to do work to help make it possible.
↕️ (:up_down:): I’m neutral
🔽 (:downvote:): I don’t support this action
⏬ (:double_down:): I don’t support this action and am willing to do work for some other sort of solution.
👍 (:+1:): General “I agree with the sentiment” message, but perhaps less specific than the upvote arrows.
✔️ (:check_mark:) and similar: I have done whatever was in this message.
🧾 (:receipt:): I’ve read the message and acknowledge receipt but no other comment yet.
🛠️ (:working_on_it:): I’m working on this now.
Anything else means some general sentiment, usually positive, but no specific meaning to me. You can figure out some sentiments by what it is and/or how (in)frequently it’s used.

For the voting things, it’s something I once read was used in some other open source project, but I can’t find any trace of it. The system was based on a {-2, -1, 0, +1, +2} to be able to express not just “good idea” but “who has time/willingness to do something”. A human needs to do the final interpretation of the results. Maybe I should write about this elsewhere.

Summary

When you use Zulip chat, you get a ton of information, but you have ways to filter it if you need. I think this also gives good possibilities for passively following along, which is also helps with onboarding and learning things.

My digital sovereignty actions

2026-06-17T00:00:00+00:00

Digital sovereignty refers to owning or having control over your data and digital services. This post talks about what I do.

I’m interested in this not just to get away from US-based companies, but for preparing for the internet to get more locked down and closed with things like age verification, forced identification, etc.

A basic goal is I’d like to do things that I can recommend to others. I used to self-host many things (like my own email, web servers, etc.) and still could, but I am much busier now. Furthermore, not everyone can or has time to self-host things. I would rather learn and use things that I can recommend to, for example, my parents and friends, rather than the theoretical perfect thing for a person with plenty of time. As you’ll see, I have various things at different levels of from “self-hosted” to “cloud”.

Proton services

I use Proton for my email, with my own domain. I pay for this. The custom domain setup was extremely easy (though of course I’m an expert here, but still… it’s almost doable by anyone).

With plans above the minimum, you get an incredible amount of stuff:

mail (works well)
calendar (I don’t really use)
drive (works well, but I don’t sync my computer to it)
docs (working, including spreadsheets), but not good enough for my primary use yet.
password manager (this has been an unexpectedly useful for me)
VPN

The basic benefits of Proton are client side encryption, being exclusively funded by users, mostly controlled by a foundation, and having a critical mass to continue developing services and expanding. Their goal is to make good services which are intended for wide adoption (not theoretical perfection), which is a trade-off I support. Other basics, like clear backups and recovery (which is difficult with client side encryption) are taken seriously and made usable, so that (for example) my parents would not lose all their data if they do the backup/recovery steps suggested.

Is Proton bad?

I often see complaints about Proton in the news, but in my analysis most are unfounded. They are a serious company making the hard trade-offs needed to provide a service to as many people as possible, which I think is good. I think most complaints (while having a factual basis) are either general negativity, or perhaps targeted attacks.

Cryptpad

Cryptpad is a web-based client-side encrypted drive service. It has OnlyOffice as a document suite, providing very high quality documents, spreadsheets, presentations, etc. It also has forms and markdown document support. I know crypto researchers what have colleagues who develop on CryptPad and recommend it, so that’s a good review to me. It’s run by a French company.

I don’t pay for this and get 1 GiB free. I used this when I needed competent office suites or documents I can share via a link, but I don’t use it for much else. Sign-up was completely anonymous (and you can even do basics without registering).

Home servers

I have two Raspberry Pis running stock Raspberry Pi OS. (There are other Raspberry Pi images that are designed to host home services more easily, but I could manage them myself. Also, someday I may switch to raw Debian.)

A Raspberry Pi 4 serves as an internal file server, though the files are just stored on USB flash drives so it’s not a fully developed, backed-up NAS yet. Sometime I’d want to make it a full RAID array. One USB drive holds basic shared files, one USB drive is backups. (I need to set up better off-site backups.) This one basically runs Samba with world-rw permissions, but is inside our internal network so guests don’t have access.

A second Raspberry Pi 5 serves as a web server. It has a dual M2 NVMe hat with (currently) one 500 GiB NVMe. Nginx serves as a reverse proxy and various servers run in Docker. This is accessible to the world and is in the guest network.

I got Raspberry Pis since it’s a Europan company and I wanted new things that would have a long lifetime. To increase reliability, I made the root filesystem on the SSD cards to read only (debian-read-only-root). The idea is I want to treat these like an appliance with a long lifetime, not so getting I mess with all the time.

Self-hosted Nextcloud

On the Raspberry Pi, I installed Nextcloud via the “all in one” distribution. With this, you pull one docker image and it controls the swarm of other docker images. The interface seems quite well developed and while this self-hosting is still advanced, the barrier to using it was much less than I expected. There have been some problems I have had to solve with my Linux experience (partly caused by my particular setups), which makes it hard to recommend this for everyone.

This is all self-hosted at home. I could use a VM somewhere, but I haven’t wanted to go that far and pay for that yet.

I learned that Nextcloud isn’t just a drive thing, but has a huge variety of apps built in. It has user management and file storage, but also things like a teleconference solution (what I use to talk to my parents now), chat, documents, calendar, etc. It’s basically a full replacement for Google Workspace / Microsoft stuff, and really could be used by an organization to replace those US services.

Drive: store files via the web, Android sync, etc. It can even use SMB as a storage backend, so in theory the NAS could be the file storage.
Deck (task management / kanban)
Talk (chat and video chat. I use this to talk to my parents now)
Notes
And much more.

The office suite is either Collabora (which I think is LibreOffice based), or OnlyOffice like Cryptpad. I found OnlyOffice to be better, but the EuroOffice project was started and is forking OnlyOffice to improve it some. EuroOffice is actually a collaboration among many companies that includes Proton, Nextcloud and Xwiki (the company behind CryptPad)

NextCloud had good apps for Android, and is taking over some of the basic note taking and task management I need to do (for example I am drafting this post in the Notes app). The office suite isn’t good enough for me to fully replace Google Docs yet.

So, while this self hosting is still not in the reach of everyone, this is on the easy side of self-hosting. Still, self hosting has a lot of things that can go wrong (I’ve already had to fix various things), so it’s hard to recommend this to everyone.

Desktop computers

My desktop runs Debian (“Linux”). It’s been stable for ages and is nice and boring.

Networking and routers

I have two Mikrotik network devices in my home, one router and one PoE ceiling mounted access point. Mikrotik is a Latvian company, and their device are known to be very powerful but hard to use, which is the trade off I don’t mind making. The easy setup was easy enough for anyone, and the powerful interface wasn’t that hard, if you know Linux networking (there is always the difficulty that if you can do anything, you want to and it gets complicated. The basics are easy enough to do) - and really, a huge amount of Linux networking capabilities are exposed. They can be used fully without any cloud services.

My home cable modem/router (provided by the ISP) is in bridge mode (routing/firewall by own router), so that it has no access to the rest of the internal networks. Internally, I have a networks:

primary (own devices)
guest (guests and insecure)
IoT (no access to the internet, I can access from the primary network) (not that I have IoT devices yet anyway)
VPN (this is routed so that in can only access the internet through Proton VPN, which runs on the router. It’s also exported to the wired network with a tagged vlan, that I can access from a vpn-only virtual machine on my desktop. I don’t use it for much but it was a fun diversion / good proof of concept.)
ipv6-only (with native ipv6 connectivity, and only that. I don’t use it for anything but it was educational setting it up.)

I invested in these to prepare for a future when routers might get more locked down and I need to be more self-sufficient.

One future goal is to give my home two public IP addresses: one for my home’s outgoing traffic, one for incoming web traffic.

I have a RB5009 and cAP ax.

Phone

I have a Fairphone 5 running the stock OS. This is at least a European company and doesn’t forcibly install much extra stuff. Updates come a bit slow, but the hardware seems perfectly suitable for my uses.

For a travel phone I installed e/OS on an old Pixel 3a. e/OS seems like it worked well and is an option for my main phone if I ever needed to get away from Google.

To talk to people I only use Signal. Yes, it is US-based but it is run by a nonprofit and has the right balance of usability and security. If this ever goes down, I will have Nextcloud Talk (self-hosted) (the chat function) that I can use for the people closest to me, even from phones.

I’d want to improve phone stuff later, but it hasn’t been a focus so far.

ToDo

I need a better backup solution. The best I am thinking is some client-side encrypted backup to a public cloud cold storage object storage. Backups and disaster recovery (or lack thereof) is a major risk when people try to self-host things.

A vision of Nordic Scientific Computing

2026-05-16T00:00:00+00:00

I was asked what is my vision for Nordic collaboration in scientific computing support. This is something I have been thinking about for years, so I thought I would write it so that everyone can see. In short, there are close collaborations, but we need some down-to-earth events to bring it together at the hands-on user-support level.

What are we aiming for?

Research is important for society, and thus computing and research is important. Often times, academics get all the attention, but us backend people (who I broadly call “research engineers”) are quite important too. Research infrastructure support is important for research, but often doesn’t get that much attention. It is valuable skill that needs practice, and isn’t taught in “school”. User support is equally important, and even less taught in school. Both need practice and a community, just like academics need to do their job.

My particular interest is scientific computing - broadly defined to include data, HPC, coding, and so on. These lessons also apply to other research engineer type fields, and we can use the lessons there, too.

My idea

I think that NeIC is a good model and about the right size. There is enough critical mass, and distances and cultures are close. NeIC has good collaboration for its projects, and the idea should go beyond the accepted projects.

Teaching collaboration

CodeRefinery has a great teaching collaboration going. I think this should continue and be enhanced. Some of the benefits are:

It is the level-1 RSE training. Most of the RSEs I have hired have attended CodeRefinery and for some of them it inspired their career choice.
Teaching is a concrete activity that we can do together. CodeRefinery team-teaching provides high amounts of interaction between instructors, so that they can learn from each other, too. Open materials suitable for reference later on are also a valuable output.
Team teaching (including helpers, notes Q&A, and local breakout rooms) for advanced topics provides a way to spread the knowledge and promote staff development.

Teaching actions

Expanded in-person breakout rooms for CodeRefinery courses.
This turns into more advanced (beyond CodeRefinery level) joint courses, which serve to save time teaching + share knowledge across countries. Teaching (even with a limited audience) serves as a way to connect.

Conferences and meet-ups

I want to get back the magic of my first NeIC conference (2017), which I felt was very much focus on hands-on mechanics of computing. Part of this is remembering the past more fondly than the present, but I think there was something there.

I want conferences and meet-ups that are not top-down (manager or funding focused) but bottom up (what are our actual problems? What isn’t known?). Also present things which are problems or in-progress, not nice-looking solutions.
I want a special focus on usability and users, not a race for the biggest or coolest systems.
Basically, I want more things like the Nordic Basic Scientific Computing meetup I helped organize. I can do this by helping NeIC conferences, or I can continue NoBSC myself.
- The key points of NoBSC were: present problems and in-progress work, not just completed solutions.

Meet-up actions

I will continue to organize NoBSC in 2027. I plan for 2027 to still be focused on Finland and 2028 to focus on Nordic expansion, but if there is interest the expansion can already happen in 2028. (In all cases others are always invited, just not with an explicit advertising focus)
I would happy to drop that and instead focus on NeIC conferences and meetups to bring my ideas there, instead.

Joint SciComp support

It’s hard to build joint support structures, since it’s hard to do free work for other organizations. It’s hard to fully transfer a support case to someone else. However, it is possible to ask for and give advice freely.

The two points above (teaching and meetups) promote a network that lets us know who to refer to. This is like the NeIC model, but broader, not just those employed in a NeIC project.

Support actions

I would focus on the two points above. It makes a network, and after there is a network where we can ask each other for help.
As part of the teaching + meetups, I would have a focus on usability and user support. This will help with internal support. (I have to do user support training when we hire new people, and would be happy to share this.)

Hopes for new IT compliance

2026-05-11T00:00:00+00:00

I recently saw that due to “organizational change”, our small-scale development IT compliance process is changing. As usual for these matters at my organization, security by obscurity is the rule (even when it’s not security related), so I won’t go into the details. But I do have thoughts, and writing lets me give suggestions.

Not all details are known yet, so this shouldn’t be seen on commentary of any decisions, but hope of what can come next in the best possible case.

What is compliance?

Let’s not be overly-complicated here. I don’t aim to go in-depth here, but there are various laws that you have to comply with when you do IT systems (let’s focus on personal data management here: privacy notices, legal use of the data, etc). An organization needs to ensure that anything its employees make is legal, so they set up some process for that.

Ideally, the process is as minimal and simple as possible. In some cases… it becomes much longer and harder than the original laws that were to be satisfied. There become many more friction points and bottlenecks, which isn’t that great for agility, and in fact can paralyze the whole organization. Clever organizations would make sure that the “compliance process” adjusts to the need of each thing that needs to be complied with.

It’s my general thought that ours doesn’t do this. I don’t want to go into why: if you know, you know. If you don’t know, maybe the following won’t be that interesting to you.

What happened

Roughly, there seems like there will be less central review of compliance forms and people/units need to take more responsibility themselves. This is likely only the case for small-scale development (as in research and services which do not have an impact on other services). I don’t know for sure, but I presume large scale procurement and development is staying the same.

This will be unfortunate if there are still very specifically defined requirements how to be compliant (stricter than the law says) - the narrow definitions for a diverse organization are the very origin of the problems we have had. But there is potential.

What I hope happens

Some of this can happen because of the upstream changes. Some could have happened anyway, but hopefully we have a good excuse to work together to implement them.

Delegation to units and adjustment to their local needs. There are very different needs from a unit which is mostly procuring outside services and one which is mostly developing research methods only on local infrastructure (before these were treated identically). Unit heads will now hopefully be able to adjust processes to suit the actual needs, tempo, and risks of their work.

Better separation of goals and procedures. Our legal teams will hopefully be able to specify what the necessary requirements to meet legislative requirements, rather than hide the requirements behind some process which is not transparent to anyone. This will help us to satisfy the actual needs and be able to communicate about with each other better about how to do that.

Better interactions with legal services. Instead of interactions with legal services being focused on filling some forms (and much friction caused by the difference between procedures and reality), legal services will be available for more interesting questions. I envision a future where we can proactively approach legal services with questions such as “Our goal is X, but we are uncertain how it interacts with the law, can you help us think through this?”

More local knowledge. Instead of all compliance knowledge being centralized in a few people, units will have to skill up. This way be seen as a bad thing, but it’s better than not having skills and also not being able to do anything because it takes too long to get answers. For basic things, the difficulties are not too much (and I would point out that for research purposes, all things are already documented and, in practice, delegated to the end researchers). Legal staff will have time to help with advanced cases, and local staff will be better prepared to work with them. Instead of a hub and spoke model where the only interaction is with a central team, we can talk with and learn from each other.

More sharing. I hope that, with more local responsibility, we will have more local sharing of our documents and practices. Right now, it is too centralized, and there is very little sharing, so everyone is trying to figure out things themselves. (It’s almost treated like a quiz, instead of collective challenge.) This sharing requires some work from us, but will lead to great improvements in the long term.

More time where it is needed. For projects with interesting legal questions, such as third-party data transfers, I hope our legal services will have more time.

Better possibilities for security reviews. Right now, compliance is often called security, but actually security is a small aspect, one which can not be reached without going through many other steps which are not always relevant. I hope that our security teams can produce better checklists (usable by our local teams independently) to be considered during development processes and will be there for real security advice for more projects than they were previously.

Standard operating procedures. As of now, standard operating procedures have not been allowed - everything needs separate review. With this delegation, there is much more possibility for streamlining the normal cases and focusing on the complex ones.

Don’t replicate research compliance. There is a whole other well-working process for managing personal data in research. In the past, this has been disconnected from the one for IT systems (so researchers have to figure out how to do both), even though it has the same goal and most of the same steps. I hope that research projects can use only that process.

More compliance. This may seem ironic, but as units determine their local procedures and checklists, they will be more inclined to use them. Lower cost means higher uptake. Less projects will fall through the cracks.

More data sovereignty. Since we can do small-scale development faster, we will be less reliant on purchased services from large US tech companies. We can adjust progresses to make this possible.

I don’t know if these will happen or not, but I hope we can work towards them.

The platform compliance pitfall

2026-05-09T00:00:00+00:00

Roads don’t have to accept responsibility of who drives on them. There are some standards for being safe, though. Imagine how much all construction would get slowed down if it gets bogged down in administartive work looking in detail about who goes over the road?

Even if, like me, you don’t want the world overflowing with cars, hopefully you agree that the best way to do that isn’t to excessively increase the costs of roads.

The point

In my work, we maintain various learning platforms. This is what one of their privacy notices could drop like:

Privacy notice for student data

Data contains: whatever you upload
Data source: you, when you upload
Data usage: as you choose to analyze it
Data removal: when you delete it
To update or correct data: correct it yourself

Or this, for the assignments submitted via the platform

Privacy notice for student data

Data contains: whatever your teacher requests you submit (in accordince with privacy notice for teaching)
Data source: you (in accordince with privacy notice for teaching)
Data usage: as decided by your teacher (in accordince with privacy notice for teaching)
Data removal: when your teacher deletes it or requests we remove their course page (in accordince with privacy notice for teaching)
To update or correct data: contact your teacher (in accordince with privacy notice for teaching)

As you can see from this, the we and the platform itself isn’t deciding how the data is used. Students can use in for their own work, in which case students decide themselves. For the submitted assignments, teachers decide how date is used (hopefully in compliance with the privacy notice for studies!).

The problem comes when we are required (by others in the university compliance teams) to make a privacy notice just for this platform covering everything that happens on it. In legal terms, we are forced to act like the controller of the data on the platform. For the data submitted as assignments, this is somewhat reasonable, since there is only one legal entity involved. For students, it makes no sense since legally, the closest model is that we are a data processor to the data the control themselves.

Note: I expect someone to say “no, what you describe isn’t what is actually required. Well, then please help us to avoid spending so much time in meetings related to the above. The purpose of this post is to help us realize when this is happening and change our mindset, not to complain or claim I know the best.

Note that the metadata of the platform, such as the login details would be covered under its own privacy notice.

The effect

What happens? We spend huge amounts of time trying to align things that don’t make sense together. You know how people say EU over-regulation reduces innovation? I think this is an example of it. All the goals are good: transparency of how data is processed and stored. But when you are pushing the limits of normality (“innovating”), the real world doesn’t line up with the expectation, and when you are in a risk-adverse society (because of huge penalties of doing things wrong), lawyers will slow down everything that isn’t normal-looking.

Instead of time doing productive work (our final goal or understanding the complexities of the situation), we spend time fighting ourselves..

What to do

Disclaimer: I’m not a lawyer but have been involved in these things for a while. I know the problem exists, but you need someone else to tell you if the below is possible and how to do it. Or hire someone who will make it possible.

Part of this is baked into the system and there is not much to do.

But our organizations need to spend more time trying to interpret things usefully, not make our documents look as normal and boring as possible. This actually takes skill, which I would think would be more interesting than debating documents the way is done now.

We should accept that some things are platforms and the privacy notices can be delegated to another part of our organizations. And better acknowledgment that sometimes we are a data processor but not controller (and the controller may be a private individual or a different part of the organization).

This will both be more transparent for our users (which is the actual intention after all), easier for us to document (long term, once we understand how to do it), and more interesting for our legal services who are supervising things. I’m not saying it will be easy, or even possible, but it will be useful.

While I haven’t been that involved in these “platform compliance problems”, I’ve seen this and think we should give more thought to the platform effects in the future.

Book review: First, break all the rules

2026-04-26T00:00:00+00:00

I recently read the book First, break all the rules, based on seeing it referred to in another source. It is one of these things about management, and I hoped to learn something that would help me in my job. It seemed to be a bit more practical and useful than some management books, and it was roughly true.

The idea behind it is that the Gallup organization (known for polls, but apparently they do more) had investigated many companies and these authors had ideas based on what they learned. Most relevant to me was twelve questions they found had the most correlation to high-performing workplaces. These sure seem reasonable to me, even as someone who isn’t an expert in this:

Do I know what is expected of me at work?
Do I have all the materials and equipment I need to do my work right?
At work, do I have the opportunity to do what I do best every day?
In the last seven days, have I received recognition or praise for doing good work?
Does my supervisor, or someone at work, seem to care about me as a person?
Is there someone at work who encourages my development?
At work, do my opinions seem to count?
Does the mission/purpose of my company make me feel my job is important?
Are my co-workers committed to doing quality work?
Do I have a best friend at work?
In the last six months, has someone at work talked to me about my progress?
This last year, have I had the opportunities at work to learn and grow?

They are divided into groups, and for most workplaces the bottom needs to be in place before the higher levels can be achieved. There is a good metaphor of mountain climbing and altitude sickness if you go to high too fast with acclimating:

They define questions 1-2 as “base camp” (basic fundamentals of a job).
Questions 3-6 are focused on one’s individual contribution (“camp 1”)
Questions 7-10 are focused on belonging in the workplace (“camp 2)
Questions 11-12 are focused on growth

It then makes a point that a lot of management consulting and teambuilding things try to fix the higher levels without making sure that the bases are in place.

The other major point that stood out to me was “talent” vs skills and knowledge. The book defines talent as something intrinsic about a person, basically their personality or interests. Skills and knowledge can be learned. The book goes through extensive arguments why you should accept someone’s talent as they are and not try to change them, rather than try to force people to doing what they don’t want. Basically, treat everyone as an individual and find a way for all individuals to do they best they can do while meeting your goals, even if it gets creative. Don’t try to change people, understand and adapt to them. (To think of types of talents, it says there are striving, thinking, and relating.)

This seems pretty obvious to me, especially since at one point I decided “I’m not doing academics, let me adjust to what I’m good at”. Perhaps when the book was written more than 25 years ago, it was more novel?

For selecting employees, it says:

When selecting someone, they select for talent… not simply experience, intelligence, or determination.
When setting expectations, they define the right outcomes. Not the right steps.
When motivating someone, they focus on strengths… not on weaknesses.
When developing someone, they help [them] find the right fit… not simply the next rung on the ladder.

There is some info on making customers happy, which is also relevant for my team. It says on four levels, in hierarchical order:

Customers expect accuracy.
Customers expect availability.
Customers expect partnership.
Customers expect advice.

Overall, I’m glad I read the book and think these questions are a good guide for our team, however, I’m not saying others should read it. It was entertaining enough and wasn’t that hard to read, though, and gave me stuff to think about. The things above are enough of a reminder for my future use, even if there isn’t that much different from what I try anyway.

Notes on Debian with a read-only root

2026-04-12T00:00:00+00:00

I run several Raspberry Pis for home servers. They currently use SD cards for their root filesystems (the actual data is stored either on USB disks on NVMe disks), and I wanted a way to preserve the life of the SD cards. Can I make them more like an appliance, that just works and only rarely gets updates, rather than a normal server that’s always writing to disk.

This was tested with Debian 13.4 but probably works even for much older.

The basics

Basically, add ro to the mount options for / in /etc/fstab:

PARTUUID=7ad6db44-02  /   ext4    defaults,discard,noatime,ro    0 1

This worked enough for the computer to boot and me to ssh to it, and my feeling is it’s hard to get locked out enough that you can’t ssh to it.

When you need to go to read-write mode, you can mount -o remount,rw /, do what you need (package upgrades, etc.), and then reboot (or you can try to remount to ro, but it might not work if some files are open rw).

Stuff that automatically works

The following are automatically mounted as tmpfs by default (among other things) * /tmp * /run * /run/lock * /var/run * /dev/shm * /run/user/0 (per-user tmp)

What is also already set up on a Debian system:

/etc/mtab (a symlink to /proc/self/mounts)

Systemd / journalctl seem to track recent logs in memory, so it just works as expected (you get logs from the last boot, plus logs that were written to disk before the logs dir weth read-only).

Things to do

Install resolvconf and it will move /etc/resolv.conf to /run/resolvconf/resolv.conf. For certain network tools you need to configure them, but it seems that for all the default stuff, it just works.

Samba

A samba server will fail if it can’t open certain directories read-write. For some, you can mount a tmpfs there via /etc/fstab:

tmpfs    /var/log/samba         tmpfs   defaults,size=10M   0 2
tmpfs    /var/cache/samba       tmpfs   defaults,size=10M   0 2
tmpfs    /var/lib/samba         tmpfs   defaults,size=10M   0 2
tmpfs    /var/lib/samba/private tmpfs   defaults,size=10M   0 3

I don’t know if mounting /var/lib/samba will cause problems long-term, but in my case, where there is no client state needed, it seems to work.

Nginx

We need to make it able to open logs. In /etc/fstab:

tmpfs    /var/log/nginx          tmpfs   defaults,size=50M   0 2

This would need checking to see if we run out of space for the logs, but hopefully logrotate works frequently enough.

Docker

I have already moved /var/lib/docker/ away from the SD card to a NVMe disk on the system. Docker didn’t need anything else, but containerd needed something mounted to be able to start. In /etc/fstab:

tmpfs    /var/lib/containerd/    tmpfs   defaults,size=50M   0 2

Manifesto on university IT departments

2026-03-10T00:00:00+00:00

Alternative title: Challenging patterns with university IT for research

This was written in 2025 and first publicly read on 12 June 2025. It’s now being released publicly.

BEGIN MANIFESTO

In my travels, I’ve met many people and seen one pattern: at every university, people are upset at their (central university) IT departments. Why is this? I’ll argue it isn’t because of the people: individually, at my university, all the people I know in our IT Services unit are quite competent and motivated.

I think the problem goes deeper. Organizations have learned that IT can become just as chaotic and unmanageable as any physical objects can be. Enterprise IT has adapted to treat IT like any other infrastructure: make sure it is stable, secure, and efficient over long-term. Since IT is so critical and not simply something to be thrown away if it goes wrong, IT departments must make sure that nothing goes wrong.

Some research just needs reliable and normal IT tools to do their work, since their work isn’t about the computing. For them, an IT department is just what they need - it’s a basic infrastructure department like plumbing or electricity.

But for those of us for whom the computing/data/software IS the research, it’s different. By definition, we move fast and break things, and for us, it’s the IT stuff we are pushing against. For any impactful thing we discover, we’ve iterated through many wrong paths along the way. By definition, we do many things wrong all the time. We are collectively trying to enable “radical creativity” and “game changing” using IT departments which are designed to make sure that there is no risk of anything going wrong. Imagine every architecture student needing to go through city planning permission to make a model building for a student project - it’s just not appropriate given what the project is and the risk to society! This is what’s I think is wrong with institutional IT departments: it’s not designed for the purpose for which we are using it. It’s infrastructure, not research support.

This doesn’t work, and in fact produces the very outcome we so much want to avoid - a bunch of researchers going wild without control, since they have learned that asking for help doesn’t actually help. We want our IT related research to be secure and well managed. But we are trying to make that happen by a department which is working on a whole other level, one that hardly shares the same language with the researchers who so desperately need help. The result? The researchers (either more actively or passively) ignore the IT department and do their own thing. If you had a fixed deadline for graduating, finding your next job, or being promoted, would you not also do the same?

Said another way, research computing is not an IT system. Maybe technically it is, but, it is fundamentally different. An IT system processes data. A research system is a tool for a researcher who is the prime processor. You can spend all the time you want trying to make the system perfect and compliant, but in the end, it’s for naught, since you’ve evaluated the system, and not the use of the system.

So what to do? It’s not a problem with the IT departments. It’s not a problem with the people within them. They are good at what they are supposed to do. It’s a problem with how they are being used. We need to recognize that an enterprise IT department is a reliable base to build on, not a research support unit. Our management needs to properly separate the concerns of basic IT infrastructure and research support. Some do this by separate departments such as “research computing” or a “computing center”. If they are included in the same unit, there needs to be proper firewalling of the different concerns. If they must stay the same, we need managements who will provide a better mission for IT departments, and IT department leadership with the vision to handle both halves at the same time.

We’ve built the worst of both worlds: computational researchers who aren’t supported, and IT research that isn’t controlled or secure. We can do better. In the current world, we have to do better.

END MANIFESTO

Book review: Systemantics

2025-12-15T00:00:00+00:00

I read the book Systemantics by John Gall. I saw some reference to it from another book, and Systemantics was short, so I figured why not. In the end, I’m glad I did - while it’s not scientific book, it’s given me a way to put various problems into words, which will probably be useful.

The Wikipedia page linked above describes the basic contents pretty well, so I won’t try to give a very long summary of the contents here and instead focus on what I got from it.

Summary

The book is basically about “systems”, as in collections of interacting elements - in particular ones of humans working together. Think bureaucracies, recruitment processes, supply chains, etc. You’d think that making a good process (= “system”) to handle something is easy: make the process, apply the process. But in reality almost everything is strongly interacting with other things, and what works in theory almost never works in practice. There are plenty of anecdotes, stories, and philosophies about things.

Some of the more important points include:

A working big system has always grow out of a working small system.
You can’t just transplant a working big system from one place to another. You can’t really fix up a non-working complex system, you have to start over.
Systems always grow, and when a system gets too big it always starts to fail simply because of how complex things get when many parts interact. (Just think of how many sayings we have about things not working, such as Murphy’s Law. It’s almost like in life, we expect things to not work by default.)
Any changes tend to have unintended side-effects
“Do it without a system if you can”, then “do it with an existing system if you can”, then “do it with a small system if you can”.

Thoughts on the book

The first thing that became apparent was this book doesn’t take itself too seriously - it’s full of humor and almost sarcastic at times. Still, it makes you think in a way that is useful, just as a reader you need to apply plenty of your own thinking.

One of my most memorial examples of this hyperbole is about ships and logistics. The book gives an example of how global shipping has grown out of control, until the engineers, in their quest to expand the system, have apparently forgotten that ships need to go into harbors, and the latest large oil tankers are too large and must be loaded and unloaded offshore. Well, sure, ships grew too large, but the system adapted because this is more efficient. It wasn’t somehow forgotten. (Still, the purpose of the example is make you think about things growing without bound.)

If you can use a critical mind and look through this, you can see various insights into human behavior. I probably say it isn’t goundbreaking, and it isn’t original research, but reading it will help me to put thoughts into words, which will help me in my communication later on.

There are definitely connections to the book I read “The Innovation Delusion” which has the main hypothesis that people like making new things, but very rarely there is value of keeping existing good things going.

Relevance to my work

I work in a university that’s relatively new - it was merged from parts that are much older, but many of the processes being made are new. Those parts are being made in a more complex regulatory environment with greater demands both on the community and from society. That means we are trying to make many new complex systems.

In short, there are plenty of times research has to do something new: new laws or needs require something else (personal data protection, security, research ethics, etc.). What happens each time at my university? Let’s make a new process to deal with it! We end up with many parallel processes (systems) trying to do the same thing. Quite often, lots of work is put into them, and most of the community doesn’t even know about it (despite the community taking part being critical).

We shouldn’t try to make things perfect at the start. Start small and grow.
We should accept imperfection if it means systems are more loosely connected, it’s less bad than strongly connected complex systems which probably won’t work anyway. There’s an opportunity to grow.
Start small and usable and slowly grow to be better.
Use existing good processes and slowly expand them to cover more use cases, when the purposes are close enough.

RSE lessons from Civil Engineering

2025-07-31T00:00:00+00:00

I have had a long-running metaphor of {academic researchers, research software engineers} being like {architects, structural engineers}. The basic idea is that coming up with interesting and worthwhile ideas/designs is a distinct field from giving ideas “structural integrity” (=reproduciblity and technical rigor).

I’ve been thinking of this metaphor a lot, and decide to learn more. Thus, I read the book Civil engineer’s handbook of professional practice, by Karen Lee Hansen and Kent Zenobia (2011, first edition) (but there is a second edition). I expected for the book to give me various insights that could be useful for developing a team of research engineers. It was about 500 pages, so not exactly recommended reading for others, but I certainly got very many insights and am glad I read it.

Below, I quickly summarize each chapter, say any particularly interesting things I learned, and in bold tell any recommendations to our team based on the chapter contents.

1 Introduction

This chapter talks about the goals of the book and the education of engineers in general. For example, it talks about how accrediting organizations for universities have tried to define what should be part of various curriculum at different levels: bachelors (4-year degree) level, masters, and professional experience. These can be defined in terms of Blooms Taxonomy levels, for example.

My main take-aways are about how there are people who think about the skills which should be learned, a lot like some people have tried to define RSE training programs. The most notable thing for me is how plenty of the skills are only learned by professional experience after graduation. Somehow, this matches with my feeling: I don’t have a direct desire to make a RSE undergraduate curriculum, but people can learn basic skills of computational science during a variety of undergraduate curricula. I am currently more focused on the practical training after becoming a computational researcher or starting a RSE job.

I would now want to be more involved in RSE curriculum learning goals, but not too much. This chapter gave me lots of inspiration for how to onboard future research engineers to our team, especially those who join at a more junior level than our current staff. I think the hardest part isn’t so much about basic skills, but how to work in a team to whose mission is supporting science.

2 Background and History of the Profession

This chapter gives a history of civil engineering. “Civil” originally meant as opposed to military, though now it means construction. Before modern times, there wasn’t a distinction between architects and engineers, or military and civil engineers. There were just engineers who did it all. The engineering profession came about because construction became more complex during the industrial revolution - more specialization was needed. There is a lesson for RSEs here: computational science has become that complex that everything a person needs can’t be included in every undergraduate program.

The book mentions how the earliest engineering projects in Mesopotamia don’t survive, because they didn’t have many stones, or fuel for firing bricks, that could create long-lasting structures. There’s a metaphor to research software which is lost to time because it wasn’t made sustainable enough because the developers didn’t use the right tools.

These days, there is advanced engineering education with a strong scientific basis and many career paths.

This section makes me feel good about my architects vs engineers metaphor and gives me some new ideas to add to the metaphor. We should emphasize that advancing technology means that more career specialization is needed.

3 Ethics

Engineering ethics is important because bad engineering can harm society. (I forget which chapter I read it in, but somewhere in this book it mentioned how in ancient times, if an engineering work killed or harmed someone, the engineer would be killed or harmed similarly. That’s some real pressure to get it right.) Different professional organizations have their own codes of ethics. They are divided to fundamental principles (fundamental doctrines) and fundamental canons (broad principles of conduct). Common things they include are integrity, impartiality, fairness, corruption.

This chapter also talks about licensing some (getting a license to practice and sign off on projects as the ultimate guarantor). Imagine if academia was so tightly controlled that a professional in research integrity had to personally sign off on the scientific rigor and reproduciblity on each scientific output (under the actual threat of damage and investigation). Considering the reproduciblity crisis, this is a huge gap between the expected rigor of civil engineering and academic publishing.

It’s appealing to think of formalizing a RSE code of ethics, licensing, and signing off on results. In practice, though, I think the harm we can do is much smaller and we can’t justify being as strict - the risk reduction isn’t worth the slowdown. Academia has plenty of ethical issues, and those should be addressed within academia first (not as a separate RSE solution).

4 Professional Engagement

A lot of this chapter is about defining work and negotiating contracts, and different forms of project organization: for example, is the designer the same as the builder? Are they part of the same contract or different contracts? What are the different cost structures and selection criteria?

There’s a good lesson here about design-bid-build (designing and building done by separate parties with separate agreements with the owner) vs design-build (one organization designs and builds). There are plenty of places in science support where one organization gives advice (for free) on how to do something, but doesn’t help to do the actual task. This is like design-bid-build and the “builders” (either within the original research group, someone they hire, or RSEs) may not be in a good position to implement that advice independently. Our RSE team is more like design-build: we give advice, and we can implement it. The outcome is much better overall.

There’s plenty about writing proposals (key points: don’t forget to understand the problem, client requirements, alternatives, and other assistance) and contracts.

In our team, we could spend more time at the beginning of projects making sure that the scope of work and expected final state is well understood at the beginning - some of our projects are ad-hoc now, often diving in without fully understanding the results. We shouldn’t become too strict, though: unlike civil engineering, research needs to be far more agile.

We should learn how to write a better project plan (to confirm with the customer) that is also as short and has the least waste. Learn how to narrow down to the customer’s goals more quickly (which is harder for us since computing is much more diverse in tools than civil engineering).

5 The Engineer’s Role in Project Development

This chapter talks about how the client, architects, civil engineers, and contractors (builders) relate. Design is split into schematic design (the broad details and vision), design development (making it more concrete), and finally construction documents (what a contractor can build). A collaborative design process (client, architect, engineer) is very important.

The predesign process results in a “Statement of need”, which allows the client to internally approve the project to go forward. This states the problem, but not the solution. The sooner the needs are known, the more efficient the project will be. Coming up with the statement of need is itself an art and requires a detailed understanding of the clients needs and application of engineering thinking. The engineers give the statement of need, as the have developed it, back to the client.

The design phase has various checkpoints different fractions of the way through. Engineers can help the owner during the bid phase, help supervise construction, and can help prepare operations and maintenance plans.

A good thing about our RSE team is that we know the people we serve. We can easily know what people need (even if they don’t know) without an excessive amount of preparation, and work agile-y from there. We could do better about clarifying the statements of needs before we go off and start working, though.

6 What Engineers Deliver

This chapter talks about the end product of engineering work. Unlike in our RSE team, where we build things ourselves, in civil engineering the engineer usually delivers design documents, which someone else (the contractor) will build. Of course, there is engineering supervision during the building and so on. The end result of the engineering work is various contract documents, such as drawing, technical specifications, modifications to these, etc. Not a building.

Mostly, this chapter is too specific to apply to us, so I won’t go into details. Most of our projects don’t have a separate engineering and building phase, and we don’t have extensive design document deliverables. (We work in the design-build system, where we directly implement the end product. And our software projects are usually small enough and agile enough that we don’t make extensive design documents, for better or for worse.)

However, it’s worth pointing out that a construction (design) isn’t the only thing that engineers deliver. They can deliver reports, calculations, help with permitting, consulting to an owner about dealing with other engineers/contractors, and more. These are all things that we as RSEs can do.

We should keep in mind that RSEs can deliver more than just software. Projects and research have many components, and engineers can help with all of them. We can especially help with engineering thinking during permitting, proposals, coming up with research possibilities, and even sometimes analysis.

7 Executing a Professional Commission—Project Management

This chapter is about project management. Why do projects need management? Because they can get so complex. There’s a good 2D typology of {assembly, system, array} vs {low-tech, medium-tech, high-tech, and super-high-tech}. The first axis is for how complex hierarchically the thing is, the second axis defines how available and proven it is. The higher up the axes, the more likely it is to need dedicated coordination. Projects are getting more about diverse things working together in complex manners, instead of simple breakdowns to various specialists.

What’s a project? Something to accomplish a goal in a fixed time period (otherwise it’s operations). The project triangle is scope, budget, schedule. It’s a trade-off between them and all three have to be part of the negotiation with the client.

Who works on projects? The owner (client), designer (decides what to do), and builder (does it). Often times other organizations give advice on design without helping with the building. In a team, if’s the same project, all must work for the same customer and goal. In most of our RSE projects, the designer and builder are the same, but we do often help come up with an idea and leave it to the owner to build it themselves (“help you do it”).

Projects have phases. In civil engineering projects, the phases include definition (predesign, 10% design), design phase (schematic design, design development, construction documents), 100% design, release to construction. Each of these has a checkpoint where the design should be reviewed with the owner (and presumably others internally) to make sure it’s on the right track.

Estimates are very important. Estimates should be made as soon as possible but will be refined over time. Make sure to update them. Prevent surprises.

A project management plan outlines how the project will be organized, how everyone works together, and a whole lot of other important reference information. It’s important to have all this reference information in one place to keep everyone on the same page.

The role of a project manager is to basically take ultimate responsibility for everything: ensure the project is done correctly, ethically, client relationship, finance, business development, etc. There is also a design leader who is responsible for the actual work (who can be different from the project manager).

There is various discussion about tracking the progress of projects, but for us RSEs we have good systems using repository hosting services so I won’t go into that.

Risk needs to be managed within projects: at least mentioned and acknowledged, preferably allocated to the different parties.

Realistically, most of our projects are too small to need a project manager for their day-to-day work: there may be one RSE working, with one or a few academics. If a single project needs a dedicated manager, it’s probably of the academic side of the work. At our size, I manage the team itself, and I think the RSE staff can mostly the work within the projects (more or less independently). However, there are many considerations which we can learn from, such as structuring the various phases with checkpoints and having a written management plan that also discusses how the project itself works.

The {scope, budget, schedule} triangle, risks, and project management plan should more frequently be discussed at the start of our projects.

8 Permitting

This chapter is about environmental permitting. It talks about how engineers dislike the process, think they know better and should be independent, and on top of that have little training in managing permitting. Sound like academics? A lot of the advice applies to academia too.

Accept that permits exist for a reason (before them, there was widespread societal harm). Respect the agency staff implementing permits: they are professionals and usually engineers. If you do your research first, understand the requirements, read the instructions, etc. it will be much faster. Initiate the permit process early. Make permits a first-order task, not a backend process (dedicated permitting managers, consider permits from the bidding phase, engage with the agencies, etc.).

To streamline permits, 1) prepare a combined project description with all the basic information, which can be reused among each permit. 2) don’t skip step one. 3) Use this in each application. This also forces the advance planning, and it’s OK if the project description get updated.

However, I do have thoughts about this and our university’s processes. There are different aspects of “permits” in academics (ethics, security, etc.), and all the processes are created internally, and still you have to go repeating things over and over again for every single form. The “one combined project description” strategy doesn’t work. Also, the ideal situation of “permit staff are experienced engineers” isn’t as true within our university environment (see below).

Our university should consider how to unify processes so that materials can be reused much more than they can now (in line with the recommendation of a unified project description for all permits). Right now, there are many different processes, not unified and with a lot of repeat work. Also, unlike the description of agency staff being experienced engineers, in our university the staff are often lawyers or similar, thus they can’t engage with advising on the actual research projects. If we hired more former academics/researchers to be involved in permitting, it could go much more smoothly.

9 The Client Relationship and Business Development

This chapter is about building relationships with clients and getting work. Engineering consulting firms are always looking for clients, but luckily our RSE team is an internal team thus we have built-in customers and funding. Still, we need to keep our customers (academic researcher) happy so they come back and give us good word-of-mouth references.

The relationship is built on trust and commitment. It’s mediated by communication. That should result in understanding what the customer needs, meeting their needs (scope/budget/schedule triangle) with quality. And this should be done with ethics and integrity. It’s important to manage client expectations and not promise too much.

It’s important to keep customers: easier to keep happy customers than to find new ones. However, you should also be careful about what customers you choose: make sure you choose ones that match what you can do and set the expectations well. Networking, volunteering, and other engagement helps to find customers.

Then there’s business development, which is the work needed to find new business. All employees do this to some degree, but it larger companies there may be some dedicated to it. You find leads, do background research, decide if you want to bid, and prepare the bid, and hopefully get selected. Coming back to the beginning, since we are an internal team, we are often selected by default since we are “free” or “at-cost”. But we shouldn’t forget the work needed to maintaining these customer relationships.

Conflict management with customers: collaboration, compromise, co-existence, capitulation.

Our team is viewed excellently by our customers, but we cheat since we work for free or below-cost as part of research support. Still, we should all keep in mind the customer relationships. We could be a bit more picky about projects and make sure the expectations are set at the beginning and not promise too much. We should engage with the broader scientific and user community to make scientific computing and our services better known (rather than focus only on direct marketing).

10 Leadership

This chapter was a bit more about business leadership, rather than project management discussed earlier. This proposes that leadership quadrants are (business strategy) (business economics), (technology), and (public affairs/marketing). Leaders plan, organize, lead, and control. Plenty more has been written about leadership outside of this book and I don’t think there’s much more insight here.

As our team gets bigger, I need to think more about how it’s led and how I can make sure everyone can have their voices heard. I think I’ve been doing better here lately, and think that many of these broad visions things will occupy the next months or years of my time. We can start thinking of different types of leaders within the team.

11 Legal Aspects of Professional Practice

This chapter goes into details about the effects the legal system has on professional engineers. We are fortunate to not have to worry about that too much, being focused on research with less stakes.

There are some good thoughts about concepts such as negligence, liability and warranties.

It talks a lot about engineering contracts and problematic contract terms. It also talks about the main contract types, design-bid-build and design-build, but there are even more (consultant to the client separate from separate designer/builder, multiple prime, construction manager, etc). Then different types of contract, fixed price, cost reimbursable, etc.

Then there is risk management. You want good notes on everything to prevent “you never said that…” situations. One particular point is that it’s good to think about risk allocation (who bears the risk) before you start, and it’s usually best that risk is borne by those most able to minimize it or be aware of it. Then a bit about insurance. Then there’s stuff about dispute resolution (litigation, alternative dispute resolution). Then anti-discrimination laws.

We are fortunate that we work within one organization, so that we don’t have to deal with most of these legal aspects. However, there are good lessons about keeping good notes (which can help conflict resolution later) and risk management (make sure it’s discussed). Finally, it’s worth thinking about the (internal) formal relationships: are we design-build. Or design-consult? Or is it a multi-party relationship? This sets the expectations.

12 Managing the Civil Engineering Enterprise

This chapter is about finances. In addition to asking “can we do it?”, engineering firms should ask “should we do it?” with a go/no-go decision process. This can ask how well the work is understood, client’s expectations, degree of profitability, and risk factors.

There is plenty about costs of labor, overheads, multiplies, financial reporting.

It also talks about human resources: who’s available for what projects? What’s their costs? What’s their career path and targets?

Then there is business development: finding clients and all, which includes marketing and being involved in the community.

Our team is fortunate to not have to worry about per-project profit, since we are also paid to maintain overall research quality. However, I think a “go/no-go decision” would be useful to us as a way to make sure we understand everything before we dive in (even if we always want to give some sort of “go”, even if it’s smaller than requested). And outreach doesn’t have to just be marketing for clients but can be general engagement with the community.

13 Communicating as a Professional Engineer

The chapter starts with a good general rule: “no surprises!”.

Communication isn’t just sending a message, but the reception and understanding. When communication is important, the sender should ask for an acknowledgment (preferably including a summary as they understand it). When things are important (for example, spending huge amounts to construct something), there is no room for mistake, and it is the engineer’s responsibility (not the client) to make sure that communication goes through. Don’t forget the importance of nonverbal/nonwritten communication (tone of voice, body language, etc.) Efficient communication is needed to stay profitable (it saves time and improves the work).

It talks about how to use email. I guess since the book was written, this is now much more known, but there are some good points: be clean, In the last 12 years this is sort of taken for granted. Use good subject lines.

Conflict resolution was also included here. Conflict resolution strategies include collaboration, compromise, coexistence, and capitulation.

It then talks about report formats and so on. As described in previous sections, engineers have to produce lots of proposals, responses to proposals, plans, etc. that aren’t in use by our team. In some ways, that’s good for us, but there are some places where better writing and planning could help us. It’s good to make sure to include executive summaries at the beginning and recommendations at the end of each document (most people don’t have time to dig through to find this or figure out what’s going on).

One thing that struck me is how formal engineering communication can be (with the idea that, you have to be very sure that you don’t get miscommunication or people remembering things differently). For example, it showed example of “phone call” and “meeting” forms, to easily make written records of verbal interactions. This can prevent surprises later.

I think communication should be one of our focus points. With users, we should make sure there are good records of what we discuss and the plans we make. We now have simultaneously editable web documents, which is extremely powerful compared to 2011, and we should use this more for project communication. We should remember that time is limited and communication needs to be clear and direct. These days, when I send emails, I always start with what I’m asking, and then go into background.

Having “meeting record forms” is overkill, but I do think there are some good lessons about being more clear in our communication with our users. Perhaps we can better use the online docs and continually add notes of each meeting/interaction there.

I have lots of other thoughts on communication from my other interest in learning about accidents, but I’ll talk about those elsewhere.

14 Having a Life

This chapter talked about work-life balance. One example it gave is that someone observed that, of their company’s executives, most were divorced and hand to pay more than half their salaries to their former partners. The executives’ take-home salary was less than a starting engineer. The point is, maintain work life balance, avoid burnout, etc. or else you will pay far more later.

It gives the key components as mind, body (eat, sleep, exercise), and spirit (feelings and thoughts). For the mind, it’s important to manage stress properly.

I’m happy with how our team works together so that everyone can disconnect during vacation and work the right number of hours, but of course a lot of that is Finnish working culture. (I was entertained by the irony that as I was reading this chapter, I was technically on vacation.) The academic system doesn’t contribute to work-life balance, and I hope that better RSE support of academics will help the burn-out problem some, by making the work less overwhelming for individuals.

One of the best parts of our team is we get academic-quality people but provide a much better work-life balance. We should make sure this balance continues in the future, and properly manage projects to prevent them from affecting our lives.

15 Globalization

Globalization is the reduction of barriers of mobility of goods, capital, services, and labor. It’s more than just mobility: it is mobility, simultaneity (things are available everywhere at once with less time lag), bypass (multiple choices and being able to bypass current structures), pluralism (not just one center dominating things).

Interestingly, this chapter is the one that talks about the effects of climate change (with the next being more about lifecycle cost analysis). Among many things that don’t need to be restarted here, it talks about how better project management and teamwork will be needed to survive in the 21st century.

I don’t have any clever take-aways for us from this chapter.

16 Sustainability

I guess compared to 2011, we now think much more about sustainability. Many things in this chapter are now well-known, or at least you probably won’t get much benefit from me repeating them. Most of the chapter is about design for sustainability.

It estimates that construction costs can be 5-15% of a facility’s lifecycle cost, design 1%, and operations and renovations most of the rest. Better design is the best way to reduce overall cost (and environmental impact). Likewise for software, more work at the beginning can result in something much more usable later. Systems thinking is needed to improve sustainability. Lifecycle costs analysis include design/construction cost, maintenance costs, and residual value at the end of the study period.

RSE already uses software sustainability as a marketing point. We shouldn’t forget this when talking about our benefits.

17 Emerging Technologies

This chapter isn’t just about new technology, but how the way technology and all interacts is getting more complicated itself. How it’s going from “individual processes computerized/automated” to “whole systems computerized/automated”. A lot of this is about data exchange between systems and having a complete, up-to-date, view of the designs and projects.

Another thing it talks about is “integrated project delivery”, basically different teams working together more closely than otherwise. Instead of relationships defined by contracts and responsibilities, a combined team is formed.

It concludes a bit thinking about “engineering thinking”: how to use knowledge to make new innovation, and how new innovation makes more new things possible.

Our RSE team by definition is often helping others be on the forefront of emerging technology, and we are often thinking about how we should adapt to the next thing.

Blog - Posts by Richard Darst

Book review: The Unaccountability Machine

Summary

Overall thoughts

Chapter-by-chapter analysis

Part 1: The nature of the crisis

(0) Introduction

(1) Something’s up

(2) Stafford Beer

(3) Aliens among us

Intermission: Computing ponds and rabbit hole

Part 2: Pathologies of the system

(4) How to psychoanalyze a non-human intelligence

(5) Cybernetics without diagrams

Intermission: decerebrate cats

Part 3: The blind spots

(6) Economics and how it got that way

(7) If you’re so rich, why aren’t you smart?

Intermission: Meanwhile, in Chile

Part 4: What happens next?

(8) Enter Friedman

(9) The morbid symptoms

(10) What is to be done?

News channels in chat

What is clickbait?

Basics

How-to

The benefits?

“Code” vs “wiki” (vs “book”) rules for project contributions

Code rules

Wiki rules

New one: book rules

Summary

Examples

Debian with unionfs root

The idea

Filesystem mounts

Network bridge to VPN

Zulipchat guide: managing too much information

The fundamental contradiction of chats

Zulip basics

Notification levels

Zulip conventions

Strategy: if you follow selectively

Strategy: if you follow everything

Strategies to follow what’s important to you

Strategies to ignore what is not important to you

Emoji reactions

Summary

See also

My digital sovereignty actions

Proton services

Cryptpad

Home servers

Self-hosted Nextcloud

Desktop computers

Networking and routers

Phone

ToDo

A vision of Nordic Scientific Computing

What are we aiming for?

My idea

Teaching collaboration

Conferences and meet-ups

Joint SciComp support

Hopes for new IT compliance

What is compliance?

What happened

What I hope happens

The platform compliance pitfall

The point

The effect

What to do

Book review: First, break all the rules

Notes on Debian with a read-only root

The basics

Stuff that automatically works

Things to do

Samba

Nginx