I wanted to write this post earlier than I, in the end, do. But as it is with many things, it’s better late than never.
This year was fruitful for me—I got a lot of ideas while working on programs, both at my daily job and as a hobby. But what makes the year fruitful are, of course, not the ideas themselves, but their execution. Not only did I manage to release two open-source projects, one of them also gained sufficient traction so that I could call it “notable”.
I decided I want to share the experience of building open-source software. It’s interesting not only as a solitary activity, although programming is often defined as being one.
So, to start a project, one first has to have an idea of what to automate, simplify, or simply implement.
My first area of interest was build systems— I had to re-implement from scratch and then grow and maintain a build system for several interdependent libraries. As one of the libraries was a C run-time library for ARM native tool-chain to use, building was not straight-forward. A lot of thoughts came to my mind as I gardened the build 1. Not all of them were baked enough to introduce implementation of them to build of a project, nearing its production use. Several loosely connected teams used this build, so taking chances near the deadline wasn’t our choice.
So we ended up with a big and messy build system written in Make mostly by one person. That made it nearly impossible to try to simplify anything of it “in-tree”—as even being the only person understanding most of it, I already started to forget the details. Besides, many decisions were made without me, and the project was extended by fellow programmers—in a way that was possible given short deadlines and possibility to understand convoluted contraptions of Make build.
At the same time, I saw a lot of possible improvements, would the build be implemented from scratch, again. Even without changing the technology itself. GNU Make, as old and complex and an-hoc it is, is still a powerful general-purpose dependency tracker, that can also be used to automate deployment (we needed to install stuff to separate devices), testing and even debugging.
So I wanted to try to squeeze every bit of goodness the old program could give. I started to poke around several prototypes.
The name—Qake—came a bit later as something resembling “Make”, but all other “first-letter-replacement” names (such as Bake, Cake, Rake, Fake) were already taken. So I took this, and invented the meaning—I attempted to cause a “quake”, as in “earthquake”—and destroy the Make’s backwards compatibility that was getting in the way all the time.
I’ll tell you soon what came of it, but first I wanted to describe the second project, too.
At some point of learning Rust 2, I decided I want to implement something that would be slightly less contrived than calculating the factorial. I doubt I came up with significantly more useful goal, but I decided to implement a simple arithmetic game, running in the terminal.
Why in terminal? This has to do with enjoying Emacs and text-based interfaces—because they are ubiquitous, easy to implement and remind me of the gone era of early Unix hackers doing away with mere terminal on PDP-11 3.
So here am I—after some two weeks I got a dirty first implementation. For the sake of being a subtle kind of eye candy—in particular, a terminal-based colorized eye candy—the game manipulated colors and other attributes of terminal. Doing this in imperative fashion every time one wants to do formatted output is not very convenient:
term.attr(Bold); write(term, "I am bold"); term.reset();
So I wanted a tiny library that would allow embedded formatting tokens to be inserted, like this:
render([Bold, Literal("I am bold"), Reset]);
Asking on Rust’s IRC channel determined there’s possible some similar code in Rust’s compiler. But it would be needed to be extracted to a nice library without external dependencies first. I decided this little act of bike-shedding was useful as a learning exercise, and set out to implement the library myself.
How open source works
Running a bit ahead of train, I have to say that another two weeks later I had working implementation of what I initially wanted. I came to IRC channel and announced it. Nobody was particularly interested—that’s understandable, given vague purpose of the library, but that’s not the point.
For me, this was the fulfillment of a dream. It was a tiny fulfillment—of the size of the project, but nevertheless.
You know, how it’s often being said:
– You need something in an open source project? Why not build it yourself?
And I did exactly that—came and asked, then went away and came back, having what I deemed useful, implemented.
This was for the first time for me. I never participated in others' open source projects. But now I understand how some projects or features of projects get the reputation of not user friendly, barely documented, fragile programs with little practical use.
Someone at some point in time thought it would be useful, implemented it, and contributed. The contribution might have been rejected if the project has strict policy on documentation or tests. Whether it should have been rejected is out of discussion for now. I don’t claim this is precisely the case, but the thought seems plausible.
Enthusiasts work on things they find interesting, and some specialists find obscure complex features infinitely more interesting to implement, than something a user would actually use and find practical. This is an interesting paradox, I believe.
What I learned from Qake
Qake, a GNU Make based build system, attempted to be a lot more practical, than terminal coloring :) I thought that it might be even useful to GNU Make maintainers—to know how their program is being used and what problems do users encounter. It may seem as strange thought, given that Make is nearly 40 years old.
In the end, I posted a link to Hacker News on first release. Interestingly enough, my own post didn’t get up-votes nearly at all. But someone else posted the same link some 12 hours later and it gained around 50 points with several comments, both substantial and not. I got two pull request during the first day, one of them adding OS X support, but these were the only PRs I got.
I also wrote a long email to GNU Make Help mailing list about what would be cool to have in Make so that implementing stuff like Qake would be easier.
No maintainer ever replied to that. I think there’s some degree of cynicism, especially given that I wasn’t a frequent poster to the mailing list. Decades of usage probably already discovered all the points I made—and maintainers just couldn’t do anything in particular about them, because of, mainly, backwards compatibility. There were some backwards incompatible changes between versions 3.81 and 3.82 already, and that cost the 3.82 a lot of adoption. Many distributions continued to use 3.81 for many years and many still do. 4.0 reverted these changes. It seems it’s impossible to challenge de-facto usage of three dozen years old program.
I think that many people already did what I did, because the one reply I got in that mailing list, is from another Make user. He said he did a similar thing at his previous job—and that’s pretty much the same story as with me. I also initially got the urge to implement these things at work, but at work, I had no time to do that. So there are probably hundreds of tiny project-specific advanced build systems, based on Make. The point being, we can never know, because not a single one of them is open source. So we have dozens of ad-hoc solutions precisely because there’s no single central repository to collect code and no community to discuss the implementation.
There probably were many attempts to improve the usage of Make-based build systems, but they mostly happened on proprietary projects.
And we now come to the purpose of GNU itself, as a project and software license. It’s viral exactly for the purpose of sharing every attempt to solve problems, implement some feature, or fix a bug. When forced to share the modified code, implementer will at least make their—maybe unfinished—work the property of community. Community then has a chance to improve it. Or in case nobody goes on to better the unfinished contribution, it’s at least there for people who consider to maybe do something similar. If a failed attempt is open sourced, everyone can study it and discern the reasons of failure. Then they may be able to mitigate these conditions and solve the problem better, next time.
This post is already too long, so I have to wrap up.
Contributing to open source development by starting a project of my own taught me a lot. It was a great experience even though there’s no activity on the project currently.
I now value the viral nature of GNU slightly more—or at least have several more points to consider why it might be a good discipline to develop software.
It also now seems a bit more practical to consider someone’s open source contributions during job application. It shows the attitude of programmer more than particular technical decisions done in the projects.
I think people who never started an open source project or participated in others' ones should absolutely give it a try.
It teaches a programmer a lot.
“Gardening” here refers to programming, in its current state, being similar to gardening more than to engineering, as described in “Pragmatic Programmer: from Journeyman to Master”. ↩
In case you’re interested in zero-overhead language with focus on system programming, you should absolutely give it a try, too. ↩
I refer to Thompson, Kernighan and Ritchie—people who invented Unix and C. A very good book on the subject is “The Art of Unix Programming”. It’s not very technical, but explains a lot of thinking, decisions and history instead. ↩