Motivations and Blogging

An Analysis of the Construction of a Linux Distribution as a Journey and the Reasons for Sharing it

Long-format blogging, rather than microblogging, is how you convey something complex that you find important enough to share. There is much that cannot be expressed or explained in few words, perhaps accompanied by a graphic or link. However, as the size of posts increases, the audience decreases. That isn't something to worry about. Those fewer individuals who stay are looking for more than the momentary amusement or bite-size insight - they want to watch the journey and are excited by the details. For anyone who finds themselves reading this, the details start here.

Let me begin by saying what I think are poor reasons for starting a distribution. You shouldn't start a distribution because you think you're right and everyone else is wrong. Not only is this likely not the case, but you've only signed yourself for hardship and isolation even if it's true. Changing defaults is also a poor reason. The information contained within your changes is better put in documentation somewhere so that people can understand what was done and implement a subset of it themselves. Unless you're being paid for it, there is only one reason to willingly throw yourself into this situation. You genuinely enjoy all of it - the stumbling blocks, the learning, and the hard decisions. That's why I'm here. The goal isn't quite doing it "right," whatever that means, it's making interesting choices that lead to a better understanding of what it is I'm even doing.

As far as the initial construction of a Linux distribution goes, the LFS book is the natural starting point. However, not listening to it lets you learn a lot more. My first changes were to the toolchain. Instead of using glibc as is expected of most anyone, I decided on musl. Instead of gcc, I went with llvm. I used the instructions from: http://wiki.musl-libc.org/wiki/BuildingLLVM, but they aren't made for a cross-compile environment. They're instead assuming that you're trying to install llvm on an existing musl/gcc system. First, I built libunwind. Doing an ldd on that reveals that it is still linked to gcc libraries from the host system - or rather, everything from the host system because the compiler has not yet been changed. This already says that it will need to be recompiled later in this process. Next is building llvm against libgcc/libstdc++. This was fairly straightforward, as is compiling libcxxrt and libcxx. Once we get to building libcxxabi, we need to add "-isystem /tools --sysroot /tools" as arguments to avoid unnecessary dependency on the host system. We also need to rebuild libunwind the same way so that the things that depend on it (read: everything, because it is replacing libgcc) don't depend on the host system either. Finally, after rebuilding libcxx against libcxxabi and rebuilding llvm + clang against it (of course using isystem and sysroot) then we have a working toolchain that isn't dependent on gcc or it's libraries. I also changed the patch against llvm to only use libunwind when compiling C++ code, reducing unused dependencies for C programs.

The next step is to be able to chroot into this new system. It isn't anything more than a toolchain at the moment, so we must fix that. I followed: http://www.linuxfromscratch.org/lfs/view/development/index.html, while using CC=clang CXX=clang++ when running ./configure and everything went well until compiling ncurses. Ncurses 5.9 has a bug that gets past gcc but it fails with the more strict compiling of clang. You need this patch: http://lists.gnu.org/archive/html/bug-ncurses/2011-04/txtkWQqiQvcZe.txt for it to compile. Also needed was CXX="clang -isystem /tools --sysroot /tools -lc++" because I never changed llvm/clang to expect headers and libraries to come from /tools. I decided not to try to fix this and instead worked around it until I didn't have to worry about it anymore.

I started with the regular tools (util-linux, findutils, coreutils, etc) and statically compiled bash and coreutils because they were referencing /lib/ld-musl* instead of /tools/lib/ld-musl* for the libc, so they couldn't run. Fixing that, everything runs in the chroot. Then I had a crazy idea. I already started with musl and llvm, so why don't I push the envelope further? I started using obase, OpenBSD userland, ported to Linux (https://github.com/chneukirchen/obase). It's an incomplete port, so I needed to patch it a good amount to have enough utilities working, and once I stop being lazy about it, I'll have my own repository with my changes. Unfortunately, I couldn't get some things to work, like "tail -f" because it depends on kqueue, and filesystem detection for "find." All in all, it's still a very usable system, and those problems can be fixed later. I also replaced binutils with elftoolchain and found myself in the process of creating a Linux distribution with a permissively-licensed core userspace. Not because I have any attachment to that idea and the implications of different styles of Free/Libre Softare licenses, but instead because it gives a great opportunity to tell people about more obscure projects. I had settled on Arch Linux's Pacman as my package manager, but I think I'll switch to Void Linux's XBFS and even remove bash from the core install. That's sure to shake things up a bit.

At this point I've got a working, self-hosting, chroot. After a few more changes, I'll explore getting this system to boot.