INTERVIEW REBUILDING FROM THE GROUND UP We meet Lennart Poettering, the lead developer of , an ambitious (and controversial) attempt to rewrite the Linux base system.

ew pieces of software in scheduled actions (like Cron) and history have been so fiercely much more. Almost every major Linux Fdebated as Systemd. Initially distribution has adopted Systemd, but a replacement for Sysvinit, the there are still some unhappy campers boot scripts that start up a Linux out there, so Mike and Graham installation, Systemd has grown into ventured to Berlin to meet Systemd’s a hugely powerful – and sometimes lead developer and get his view. complex – replacement for the “bag We also looked beyond Systemd, “Most people who say of bits” that make up the Linux base and Lennart’s intriguing proposals Systemd is un-Unixish have system. It’s growing all the time for a new packaging system to make and now handles logging, device life easier for third-party application no idea what Unix is like.” hotplugging events, networking, developers…

Systemd has now been different maintainers, different coding working on Systemd, many of us who adopted by pretty much every styles, different release cycles, different worked on the lower levels of the major distribution, and yet whenever maintenance statuses. Much of the realised that Sysvinit it gets mentioned in discussions on Linux userspace used to be pretty badly was not going to be the future. And the web, flamewars erupt. What do maintained, if at all. You had completely then I was playing around with writing you think are still the biggest different styles, the commands worked my own init system, which had the misconceptions? differently – in the most superficial funny name Babykit, and that was 10 Lennart Poettering: There are many level, some used -h for help, and others years ago or something. And then different misconceptions. Something --h. It’s not uniform. Canonical’s Scott James Remnant you always see is the claim that If we put a lot of the glue in one started working on a new init system Systemd is monolithic – and another is repository, it’s not all the way towards called Upstart. He made it public, and I that it’s not like Unix. The Unix Unix, but it’s half way between stopped working on Babykit. misconception is a pretty interesting traditional Linux and traditional Unix. We, at that time, thought: OK, Upstart one, because most people who say We do not put libc and the kernel in the is the future! Scott understood how init Systemd is un-Unixish have no idea same repository, just the basic things. systems work – it needs to be dynamic, what Unix is actually like. So that’s a misconception that I’m it needs to react to events, and it’s not What’s typical for Unix, for example, always bemused about, and I’m pretty the static thing that Sysvinit was. So we is that all the tools, the C library, the sure that most people who claim that thought that was the way of the future, kernel, are all maintained in the same have never actually played around with but as it progressed, we realised it repository, right? And they’re released in Unix at all. probably wasn’t the future, because we sync, have the same coding style, the realised that conceptually, it was the same build infrastructure, the same Another issue is: some people wrong design. release cycles – everything’s the same. see Systemd presented as an The way Upstart worked is that, as a So you get the entire central part of the init system replacement, but now it’s programmer or admin, you write: if A operating system like that. If people doing X, Y and Z on top. You’ve said happens to B, or X happens to Y, do a claim that, because we stick a lot of it’s about replacing a “bag of bits” certain thing. But we believed that an things into the Systemd repository, then with an integrated suite of tools. init system should work the other way it’s un-Unixish, then it’s absolutely the When you started Systemd, was it a around, where you say: this is where I opposite. It’s more Unix-ish than Linux case of saying to you, “we want to go to, and you figure out the ever was! want a new init system”, or… rest. Because of that design, Upstart The Linux model is the one where LP: No, it was actually the opposite. was very simple, but it put a lot of you have everything split up, and have Back in the day, when we started complexity on admins and developers,

42 www.linuxvoice.com LENNART POETTERING INTERVIEW

wasn’t moving ahead. And then we started working on it, pulled out the old Babykit code, gave it a new name, and started proposing it. A lot of people understood that this was the better approach. It was a lot more complex than Upstart – to make it clear, I think Upstart actually has its benefits. The source code is very, very nice, and it’s very simple, but I think it’s too simple. It doesn’t have this engine that can figure out what the computer is supposed to be doing. So we started writing Systemd, and Red Hat didn’t like it at all. Red Hat management said: no, we’re going for Upstart, don’t work on that. So I said, OK, I’ll work on it in my free time. Eventually Red Hat realised that the problems we solved with Systemd were relevant, and were problems that needed to be solved, and that you couldn’t ignore them. Then we convinced the Fedora Technical Committee to adopt it, and then Red Hat internal management accepted it for RHEL, and we managed to convince every committee that because you actually had to write down that Canonical tried very hard to stay in mattered, bit by bit. It was absolutely all these rules. It wasn’t the computer control of it. They made sure, with not that Red Hat told us to work on it that figured out what to do. copyright assignment, that they made it – we had to convince them. We thought: if you want to solve this really hard to contribute, but that’s what properly, then you need to let the Linux actually lives off. You get these I don’t think many people know computer do these things. And this had drive-by patches, as I would call them, that! lots of different effects: for example, where people see that something is LP: This is something that people in Upstart always maximised what broken, or something could be general don’t know. They assume that happened on the system, while we improved. They do a Git checkout, do Red Hat is this one entity, that has one think you always have to minimise what one change, send you it and forget opinion and pushes one thing. It’s really happens. And the reason for that was about it. not like that. The people who work at simply because, if you specify exactly Red Hat, the engineers, they come from what state you want to end up in, you And you never see them again! the community – they first become can pull in all the dependencies LP: Yeah, and this is great – famous in the community, they hack on recursively and boot to exactly that. these are the people you want to have, things, do good stuff, and then Red Hat because the vast majority of patches comes along and says, “Hey, do you are actually of that kind. It gives you this want to work for us?” “We started writing Systemd, polishing that you want. The people And when you start working for Red and Red Hat didn’t like it at all. invested in the project all the time do Hat, they don’t check your opinions at the big things, and don’t care so much the door. You can be sure that if there So I worked in my free time.” about the polishing. So these kind of are multiple opinions on one topic in the patches are what you want. But if you broader community, the very same The Upstart way is always, “if this is do these copyright assignment things, opinions inside Red Hat exist too. Inside started, then start that”. If the network you will never get those people because of Red Hat there are discussions. Red is up, you take that as a trigger to start they would have to sign a contract Hat has many different people, and NFS and things like that. It always has before they can send you something. most of them have strong opinions and this effect that you start as much as Putting it all together, we realised that convictions. possible instead of as little as possible. Upstart wouldn’t be it. So at one Linux So anyway, long story short, we Plumbers Conference, four years ago or And much of this debate came to the conclusion that Upstart is so, and I said that we happens in public, on public conceptually wrong, and it moved at should do something about it, after we mailing lists. Then you have some glacial speeds. It also had the problem saw at the conference how Upstart people saying that all this arguing

www.linuxvoice.com 43 INTERVIEW LENNART POETTERING

looks bad, compared to how people would hate us for it. We knew want to use the way Systemd loads Microsoft or Apple does things. But I we’d have to fight for a long time to get kernel modules from a static list, then bet they all have the same it accepted. you can absolutely replace it. arguments, just as passionately. We eventually realised that doing just Or if you don’t want to use some of LP: I’m absolutely sure. There was this the init system would never be a the more modern components like time when the people working on complete solution. Because if you do an Networkd, then use something else. I Microsoft Word had their own compiler init system but still invoke all the shell mean, on my laptop I even use to build Word and the rest of MS Office scripts and all the other things needed NetworkManager, because Networkd with. Microsoft had the Visual Studio to bring up the system, you’ve only doesn’t do wireless, right? Networkd is group, and the Office group, and they solved part of the problem. You’ve more for containers and servers. So if had their own individual compilers. solved one thing but not 90% of the you want to adopt Systemd, you can That’s just crazy of course. problem. So we slowly started doing absolutely adopt the baseline, which is So I don’t think that Red Hat is stuff that all the other Linux distros did, different from anywhere else; except and implemented that in simple C code that at Red Hat, because people are that was fast and parallelised. “Most people at Canonical working on open source, they have Debian had its init scripts, and Fedora much greater attachment to their code. had its init scripts, and they all kind of didn’t even realise that they So they have even stronger opinions. did the same thing, and did it differently, had commit access Systemd.” and some are better, and some are If back at the start of Systemd, worse. We thought OK: this is bullshit, the three components that I mentioned. you and the other developers let’s write this in C in a unified way, and You can keep the rest of the system had explicitly said: “We’re going to try to pick the best features of all – however, our implementation of the replace a lot of the base system”, do distributions and make a convincing individual parts is usually pretty you think it would’ve been better argument that it’s the right way. convincing, and usually people then received? Some people see it as an So it initially grew. But something to replace more. init system that’s suddenly touching realise is that there’s very little in everything else. Systemd that’s actually required. Some people see it as a LP: Initially it was an init system – it Systemd requires Journald, because requirement for Gnome… was just PID 1. We knew from the very every single service that runs on the LP: But it’s not actually a requirement. beginning what we were getting system is connected to Journald, and Some people don’t realise that when ourselves into. We knew very well that we need some way to log things during Gnome started making use of Logind, I touching something that has so much early boot. So Journald is a actually wrote the patch for that. I history, that is so close to what admins requirement, and is a requirement. ported GDM onto Logind. But when it do all day… That changing it would be a But pretty much all other components did that, I was very careful to make sure massive problem. So we knew that are completely optional. If you don’t it would still run on ConsoleKit. I didn’t

You need a thick skin to hack in open source code sometimes, especially if half of the world seems against you.

44 www.linuxvoice.com LENNART POETTERING INTERVIEW

Lennart lives in Berlin, and knows where to get great Vietnamese grub.

want to have those fights – if people Canonical always wanted to stay in Why do you think some want to continue running ConsoleKit, control of everything. For us it was an distributions managed to adopt they can. Those patches made it in, but exercise to make sure this doesn’t Systemd without any major fights, some people saw that Gnome now happen. We’re not the ones in power – and then others like Debian had very works with Logind, hence it must not the community is. intense debates and resignations? Is work with ConsoleKit any more! So of those 26 committers, there’s a it just because it’s a distro with more But that’s actually not true. And to my good chunk working for Red Hat right political processes? knowledge the code is still in there – now, but there are people from Intel, LP: probably did it the the compatibility for ConsoleKit. The Canonical… We had people from quickest way. You know, distributions Gnome team has the general problem Canonical in the committers group, all attract different kinds of people, of though, that nobody’s willing to the time during the discussion about course. If you looked at Arch Linux, it maintain it. People who want to stick to whether they should even adopt attracted very progressive kinds of the old stuff, they actually need to do Systemd. Most people at Canonical people – like power users. They’re some work on it. If they don’t, then it will didn’t even realise that they had commit progressive and want to make the best bit-rot and go away. access to these things. out of their computers. So it was easy So anyway, we tried to do these There are also developers from for them to adopt. things in the nicest possible way, but of Debian – two or three of them. Then if you look at Gentoo, for course people generally don’t example, they still haven’t done acknowledge it! There should be a Systemd Systemd as default. They used to be foundation! like Arch Linux is now – they used to be A lot of people just think LP: [Laughs] Well, we don’t want to the young people who adopted things there’s only Red Hat working make it too formal. We have this speed, quickly. But the Gentoo people aged, on Systemd. this quick pace with how we progress and they became more conservative. LP: Oh yeah, we’re a lot of people now. Systemd, and I think it can only work if And Debian is probably an even more Yesterday we had 26 committers, and we stay somewhat loose and not have conservative bunch. Debian is a really 40 people contributing code every strict regulations about how these old project, and many people from back month or so. The committers group is things work. in the old days are still active on it. So quite diverse, and for us it’s quite an But we try to make sure that it’s they have longer release cycles. And exercise in making the diversity of the inclusive. We have people from Arch Fedora always defined itself as being community be reflected in the diversity Linux, people from all the Linux on the bleeding edge, of course, so it of the people who work on it. This is distributions, big companies that do was easier. Well, not that easy – some also related to how Upstart worked: open source. We want to make sure it people don’t realise that inside of Upstart was very locked-down, and stays that way. Fedora and inside of Red Hat, there

www.linuxvoice.com 45 INTERVIEW LENNART POETTERING

After , PulseAudio and Systemd, we’re intrigued to see what Lennart tackles next… were lots of fights. So it’s to do with the potential open source contributors and ship updates. On Linux we don’t culture around the various distributions. from getting involved. have anything that’s as convincing. We And Slackware are the ultra LP: I’m in the lucky position in that don’t have a common way to sandbox conservatives! there’s no pressure on me in any way. I stuff, and the way that we ship stuff is know that a lot of people have with Deb packages, or RPMs. Do you read the comments pressures that they live under, and if It’s madness for third-party app when Systemd is being you also get pressure from the internet developers, to develop for Linux. Like, discussed on the net? Do you over some things that you do in your what do they develop against – which despair when it all turns into hatred free time, because you love it, that is distribution? And if they make that and flamewars? very disappointing for them. So I have a decision about which distributions they LP: For some reason it doesn’t touch luxury, and I know it, and I can only feel wanted to support, it’d be quite a few, me too much. I try to keep an open for people where it’s not like that. and then there are lots of versions. You mind and figure out what people I know a lot of people who’ve had might want to support Fedora 20, 21, actually think. There’s a lot of noise out enough of open source, and who will and 22, and then OpenSUSE and its there, but usually there’s some core of not participate in the communities various versions, and Ubuntu and its an argument – something that we where things get really bad. And that’s a various versions… All of those should actually be aware of. So if big loss for open source. distributions bring different libraries. people are annoyed by Systemd, So the test matrix, the combinations usually they ran into some kind of bug Something else we wanted to of software you have to test your apps or something. It might not necessarily talk about is your proposal for with, grows incredibly. That’s not be a Systemd bug, but we need to take packaging. What’s that about? something that’s digestible for it seriously. LP: It’s really about augmenting the third-party app developers. And it’s Nowadays Systemd is very polished Linux platform with a new way to really hard. The only way you can really in many ways, and the reason why it is package applications. It’s not about deal with that is to get your stuff into so polished is because we actually simplifying things or changing things the distributions. Then the distributions listen to people. Sometimes people say – it’s adding something to the will do all the work for you – they’ll we don’t listen – we do, but we just ecosystem that we were missing so far. rebuild for you, test for you and things don’t always agree. If we would just There are lots of people working on that like that. stick our heads in the sand and not care in different areas. at all what people wrote, Systemd If you look at all the operating But for upstream developers would certainly not be what it is, or have systems that are popular these days, it’s hard to get new releases found the adoption that it has. like Android, Mac OS, iOS, Windows into distributions quickly. Metro, they always have really strong LP: Exactly – you’re bound to what the You’ve said yourself that the app platforms, where they provide a distributors do, to the lifecycles of the flamewars could dissuade sandbox, a nice way to distribute apps distributions, the release cycles. You’re

46 www.linuxvoice.com LENNART POETTERING INTERVIEW

not responsible any more for your that they look at the code before – and then, if they’re sure that software – you’ve passed it on to the packaging it, and then you get a nice everything works, they can check it off, distributions. Which in many cases is stamp on it: this is good software. And and when it’s installed on the final actually a good thing, but in general it’s then you only have to trust the machine, it knows exactly the not what third-party developers want. If distribution, and not trust 100 different combination of software that it runs you look at how Firefox, for example, software vendors any more. against. Instead of some weird packages its Linux version, it’s a tarball Now, if we open this up and make it combination of software local to that that installs in its own directory. typical that you install one distribution machine, where they have real trouble The classic model, and then 100 different apps from 100 making sure it works. where you get everything from the different vendors, we need to do Third-party developers don’t want to distribution nicely packaged up, vetted something about this trust problem. do all the support for someone who for security problems, with security So that’s why we need sandboxing. says: “Yeah, but I have this old version updates – that’s a fantastic thing. But I We need to reduce the chance that of libc and it doesn’t work”, or “I have also think it leaves out all these badly behaving software can destroy third-party people, and if we want to your data. grow the Linux ecosystem beyond “Sometimes people say we what we already have, into something OK, but how do you deal with don’t listen – we do, we just where it’s actually useful for a broad the niggling little differences number of people, where we have between distributions, where don’t always agree.” markets and more apps, we need to everything has slightly different provide a way to make these apps library versions, filesystem this really custom distro I compiled digestible. locations? myself”. So we have this concept we And by digestible I mean: we need to LP: Our idea is to introduce something call a runtime, which is basically just a have good sandboxes. If you don’t get called runtimes. When third-party set of libraries with very specific your software from the distribution any developers develop their stuff, they versions. The idea is that you can install more but directly from the developers should be able to do so against one multiple runtimes at the same time. of the software, then you have the very fixed set of libraries, in very specific And then, if you have apps that require problem that you can’t trust the code as versions, compiled in a very specific different runtimes with different much. Distributions add a bit of trust, in way. They can test their stuff with that versions, they’ll run against their specific runtime and everything’s good. After our interview, Lennart pointed us in the direction of Didn’t the Linux Standards the East Side Gallery for some Base try to do something Berlin Wall exploring fun. similar for third-party devs? LP: What the LSB did there was to standardise a set of libraries, but didn’t define any specific versions. And classic Linux distributions only allowed installation of one set of libraries at a time – you could have one libc, and one OpenSSL. So LSB tried to make the best out of the traditional Linux model, but that’s not enough. There’s a scheme that we put together that’s not unlike what Android has. For example, if you develop an Android app, you do so focusing on one specific runtime, right? It’s one that Google defines, and if there’s something that’s not in this huge runtime, then you have to ship it yourself inside of the app, and everything is good. And the phones have a couple of runtimes for the different versions, and then you pick one of the versions you want to develop against – usually the newest version, or maybe an older one. So we kind of want to adopt the same scheme, but make it more pluralistic. In Linux everything is pluralistic.

www.linuxvoice.com 47