systemd: Please, No, Not Like This

I have become convinced, that I have spent too much time with the systemd debate already. I will try to refuse to do so any further.

I have also become convinced that systemd is a pretty mediocre implementation of some nice, but insufficiently thought-out ideas that nonetheless are pushed in highly aggressive manner. This aggressive push has already been clearly indicated as intentional, including the need to overcome expected resistance no matter what.

To be blunt: My personal verification step is over. I do not like systemd, its design, its implementation, how it is maintained, how (some of) its developers and proponents intentionally alienate whole groups of people, how even valid criticism of it is often declared irrelevant, and I have real trouble with the attitude of many of the people associated with its ecosystem. The whole thing annoys me more than it makes my work simpler, and the situation makes me feel like the things I value in open source and software engineering are being actively undermined.

If you are not able to accept that (and maybe why) I might feel this way, really, you are part of the problem. Because that inability and/or unwillingness to respect an opinion different from your own is a part of the problem.

I don't wish to be evangelized, nor do I wish do be convinced how nice systemd and its developers/maintainers are. That train has passed. I do understand that systemd tries to solve problems but I think the approach taken highly problematic both technically as well as politically and socially.

Thank you

There are several sides to my dislike, depending on who of "me" you ask:

Me, the software developer

That part of me is more than taken aback. I've read my share of system source code and I do not consider the parts I have read quality, maintainable code. Systemd developers don't document internal interfaces and they seem to see internal modularization as something that only lesser developers need to do. Yet even their external interface documentation is severely lacking, not properly documenting semantics and invariants. This also seems to hold with regard to implementing standardized behavior: Do what works in a narrow use case, ignore inconvenient details. As for their own designs: They seem to lean towards complexity and special cases without proper arguing their case. Use of orthogonal concepts, avoiding implicit "magic" automatisms (especially when they cannot be turned off) seem completely unheard off or actively eschewed. The result is that  --in many cases-- systemd features interact in undocumented and unexpected ways and  implementation details are unnecessarily exposed in interfaces, creating longevity and interface stability problems.

In many cases when I resort to chasing down a bug in systemd, I seem to find at least one occurrence of serious WTF. Let it be functions that expect an undocumented amount of buffer space passed in from their callers, a message send function explicitly silencing an error when the send buffer is full, shotgun parsing all over the place, asynchronous sync, blatant ignorance of (secure) language design, configuration formats and associated documentation that freely mix low level with high level directives into a mush with unexpected interactions requiring manual workarounds. Yet there are some pretty boastful claims (like "validates all input") that don't hold up to scrutiny or are even security regressions from previous behaviour. Such arrogant bluster combined with basic mistakes makes me very critical about any claim the authors make about their own system. In general, while systemd code looks superficially well written, its lack of sane interface design and modularization leaves me baffled.

Given these observations, I consider systemd to be a product of sloppy "quick and dirty" engineering, valuing addition of new features or at least the addition of their version of some concept over quality, interoperability, interchangeability, and stability. Systemd seems like the "lean startup" of Open Source. With the associated benefits (pace, features) and also with the associated problems (technical debt, interface design).

Systemd feels like a system built by people who work under the unshakeable belief that the very initial solution sketch they come up with is automatically the best solution. They often claim that there are no real alternatives to their own design and implementation, but their arguments mostly seem to remain shallow and not well-founded, a pattern I have dubbed "replacing competence by confidence". A credible discussion on why a complete, only partially compatible or incompatible re-implementation is necessary is mostly missing.

I often compare this to the Death character in Terry Pratchett's "Reaper Man" and how Death builds a tree swing. He starts with a limited understanding of the problem space and proceeds never taking a step backwards to reconsider a decision once committed to it. The obvious results are not what you want in a critical system component, since it requires the developers to pile workaround on top of workaround.

As an explanation, we get skewed comparisons that leave out basic facts or are based on information that sometimes is decades out of date, but the actual points critics make seem mostly ignored or "whataboutismed away".  More often than not, the "discussion style" reminds me of authoritarian propaganda: You are being informed of "facts" and not made part of an open discussion. Quite obviously, that toxic style triggers huge amounts of equally destructive pushback.

Add to that the tendency to water up promises that turn out to be too hard to keep. Examples: Reliably killing a process only works in certain (typical) cases, can be circumvented by default. Logging reliable metadata? Only works "most of the time", recently watered down because of the performance hit of metadata retrieval. Reliable system message bus? No, only best effort, croaks under load.  Use of PIDfile= has a privilege escalation bug, not implementing previously existing hardening PIDFile= implements relatively complex privilege checks, bypassing pre-existing use of simple user separation by the kernel. I have repeatedly found that such regressions/broken promises are mentioned seemingly only in passing, sometimes as a counter-argument on completely unrelated issues. This in turn causes me cognitive dissonance, since systemd's guarantees seem be often more like empty promises, to be broken at will. I find this both unprofessional as well has dishonest.

Yet, systemd developers have repeatedly claimed no wrongdoing on their part, and --obviously-- that everybody that has a different opinion is clearly uninformed, stupid, unfair, or has other, sinister motives. Of course, if you have a "dissenting opinion", they can always just threaten to refuse cooperation.

On the architectural side, the ideal of openness seems mostly gone. Instead, interoperability and openness are replaced by a hard push of a single, monocultured implementation with barely defined semantics that makes only very limited effort to stick to well-known-concepts. Even if there is no reason to diverge from existing syntax and semantics, systemd often simply ignores prior art.

Yet its own interfaces are frequently poorly defined and often mix multiple independent concerns into a single interface. Documentation language is often vague and specifications remain superficial and incomplete. Even more troublesome, systemd seems prone to introducing hardcoded automagic that cannot be turned off. The repeated bullheading of such features leaves me with the bad taste that systemd's use case analysises are done only with a very narrow focus, requiring others to work around its limitations.

As a conclusion, I think, with regard to open systems, systemd behaves like a bad citizen. It makes a lot of effort to cater to (large scale) administrators and packagers. But from an engineering perspective, I consider systemd a huge regression: It moves away from the concept of open, well-defined protocols, from the ideal of largely replaceable implementations of well-defined interfaces (conceptual composability) to a single, ever-changing implementation whose "expected behavior" is for large parts only provided by its (mostly undocumented) source code and that be changed any time at the convenience of the developers.

This is not free and open software, this is some people do what they see fit and to hell with everybody else that does not completely share their vision. "We are calling the shots. Shut up and do not bother us. We know better, anyway."

Me, the sysadmin

From that perspective, systemd is mostly a neat set of features under a single hood.

Mostly.

This is probably what caters to packagers and system administrators: systemd makes some things decidedly more easy than before. And might it seem really a breath of fresh air if you did not know any of the alternatives that existed before it. Nothing what systemd does is really new, but systemd encapsulates a lot of stuff behind a deceivingly simple interface.

Deceivingly.

It also has its share of unnecessary POLA violations. Documentation is scattered, incomplete, misleading and sometimes downright incorrect. And the docs do not even attempt to give you a mental model of the inner workings of the software. Their over-the-top, "this is the best thing since sliced bread" style of writing annoys me to the bone. Getting a good overview of systemd is hard and I have personally found that this is because that the developers don't bother to provide a coherent and consistent external interface in many cases. It feels like to provide a new feature, the developers have repeatedly duct taped new code to an existing system, never re-thinking their internal model (if such one exists) to provide a coherent and consistent interface. As is, systemd feels more often than not like a slim wrapper on top of kernel interfaces (without proper documentation on which kernel interfaces are wrapped), that combines multiple features but repeatedly fails to provide a coherent, composable model of itself. "Conglomeration of special cases with unplanned emergent interactions" is the best term I have come up with for this so far.

Me, the person

I find the blatant public narcissism, (sometimes not so) covert abuse and manipulation pouring out of systemd's main developer(s) and quite a few proponents horrifying. The use of thought-terminating clichés, wolf speech, proof by assertion, ipse dixit assertions, whataboutisms, shifting the goalpost, stonewalling, and similar "dirty" discussion tricks is particularly worrying. This list is extended by attempts of (emotional) blackmail "Be nice to me or I will ignore you" that should speak volumes about the personalities involved. Yet, somehow, these eminent signs of the dark triad do not give pause, but rather most people seem completely blind or in (unconscious) denial. And don't even get me started on the frequent use of manipulative, covert aggressive speech patterns.

I keep stumbling about the "pulseaudio introduction to Ubuntu" debacle, where somehow, all resulting problems are now blamed exclusively on Ubuntu. There is, however, another interesting timeline where Poettering attacked Ubuntu for being such slow adopters and repeatedly claimed to Ubuntu devs that pulseaudio was production ready. However, when the big fallout happened, blame landed purely at Ubuntu.

Of course, covert aggression is meant to achieve exactly that: It is a method to hide the disregard for the opinion of others, a manipulation tool to lull those around you (and sometimes even the victim) into the false sense that civility has been maintained. But in fact it is just a more devious method of being aggressive and antisocial. A way that seems completely unintelligible to a large set of people who do not understand even the limited subtlety of the process. I am sorry to tell you, but most destructive behavior does not announce its intent with fanfare. Instead it is surprisingly banal and thus maybe even more destructive. Only the really sucessful  abusers manage to make themselves appear (including to themselves) as only defending their own good intentions, because obviously, nobody really wants to be evil.

And, yes, quite a few "open source folks" do play this game (as do many others). That large parts of the community refuse to even consider the issue, is even more dysfunctional. White, middle aged, heterosexual men that a certain dev complains about? Yes, this is exactly their game. Mirror, mirror on the wall.

This is --of course-- not a problem limited exclusively to systemd supporters. Idiots happen everywhere, regardless of their convictions. Yet --again-- the  open disregard, the casual devaluation of whole parts of a community (e.g. "security circus") are a repeated pattern. All of these are strong indicators for pervasive destructive personality patterns. And --I might note-- tu quoque does not excuse bad behavior. You are not a nice person because others might be worse.

This also leads me to another, very much sociological issue that is by chance even worse than the primary culprits: the blind apologism or even unfounded aggression so pervasive on both sides of the debate.

I've seen this pattern before around typical abusers. They are surrounded by a multitude of what I call "asshole excusers" (more formally known as co-dependents) that keep on looking for more and more reasons why their idol is not has he/she seems and that everybody else is to blame. Groupthink is rampant.

To quote some of the "arguments" I had to read: "You may not complain, because you get this for free" (i.e. free software may not be held against requirements), "if you don't like it, write your own" (yeah, right), and "you have a choice" (there's no enterprise distribution without systemd and software dependencies on systemd are increasing) and other kill phrases are in frequent use. You may not question the emperor.

Freiheit ist immer nur Freiheit des anders Denkenden

(Freedom is always the freedom of dissenters.)

-- Rosa Luxemburg

What hits me are not the arrogance and aggression. Arrogant jerks are easy. What hits me are the repeated attempts to argue the abuse away and to silence criticism. Excusing the perpetrator is never a good sign (even if he/she is also a victim: tu quoque). Yet for many members of the community, introspection is forbidden. If you point out the flaws in the process you are a "vocal minority", "uninformed", a "whiner", a "snowflake". Sounds like Open Source Trumpland? It probably is ...

I cannot shake the feeling that the behaviors around systemd will serve as an excuse for the toxic parts of the community to feel even more empowered. Certain aggressive personalities already used the excuse "Linus curses at kernel developers, so it's okay to be an asshole" to rationalize their own aggression. With systemd, they get another excuse. Because when systemd proponents can dismiss any criticism to their own toxicity as deluded utterings, so can everybody else and claim it's okay to do so, too. And obviously, the "other side", will also dig into their trenches, because how are you going talk to people that disregard you and your opinion (without actually listening to it).

Make your own conclusions from that.

And as always, the whole situation would be hilarious, if it was not so overarchingly toxic while trying to maintain the illusion of progress. Pushing your agenda through aggressive means isn't progress, it's regression into a dark age of non-cooperation and despotism. Yes, it might get you to your goal faster, but it also burns all bridges.