So, this weekend was… interesting.

Through something of a comedy of errors, the server suffered some strange software issues that prevented it from working this weekend.  There was a bug in a recent update to the server software (running testing software) and that caused longer downtime than it should have due to various interactions between things on the server.  The good news is that this is mostly fixed.

Additionally, this downtime has taught me that there is yet still more to do in terms of getting the server able to stand back up on its own.  I’ve simplified the server’s setup a bit, and I have to write some scripts and other little glue here and there to tie down some of the things I’m doing so that the server can do things like go down and come back up without issues, all by itself.  Getting that done would be generally a good thing.  First things first, I have to figure out a decently reliable way to shutdown the system without having to do something like kill the containers and not give them the chance to cleanly shut down.  Ideally, there would be some sort of command that could be run on the system that would enable the containers to be shutdown.  This is slightly challenging, because you cannot just chroot into the directory tree that the VMs are running in and kill processes, because things like /proc inside the container aren’t visible to tools running on the host.  Oops.

So, I have some work left yet in terms of getting the server going the way it needs to be again.  After that, I’ll be working on it over the next couple of weekends to try to increase its robustness, and so planned weekend downtime for *.trausch.us and www.mischiefinoverdrive.us can be expected. 

Frequently enough, I see people forwarding messages to mailing lists that were intended to get to the ML in the first place, but didn’t happen due to some configuration issue or whatever.  There are people that think this is fine, this is workable, and it is correct.  I happen to disagree, but discussing this in the context of an existing mailing list software’s mailing list is proving to be fruitless (I would have guessed this initially anyway, but it wasn’t my idea to take the conversation there…).  So, here, I attempt to state exactly where I’m coming from as a starting point.  This is the first part of what will hopefully be a productive series of posts that strive to define what mailing list software ought to be, how it ought to work, and gradually refining detail until it becomes a blueprint for an initial implementation, at which point it would be my intention to begin a project to do the implementation.

Base Position

Mailing lists should Just Work.  Mailing list management servers are software just like any other software, and they can be made to do what is convenient for us humans, and they should be made to do what is convenient for us humans.  Mailing list software should cater to the needs and desires of all its users, regardless of technical aptitude or choice of MUA (mail user agent, email client) software.  Mailing list software should be updated to operate with the expectations and flexibility expected of software used in the 21st century, and provide more options for interaction both technical users and non-technical users alike can love and use.

It is my personal belief that both mailing list software and MUA software should follow the established philosophy of "do one thing and do it well".  That means that MUA software should be expected to only be capable of being a basic mail user agent, and that mailing list server software should be expected to be capable of everything required to manage a mailing list for end-users, technical users, and list administrators.  A finer-grained permission model in mailing list server software could be adopted to make some of the proposed features even more convenient and get rid of the "administrator" vs. "user" line (i.e., "Jack can unsubscribe users", or "Carla can list messages in the mailing list’s pending approval queue", or whatever).

Furthermore, it is my belief that enabling convenience for end-users does not necessarily mean taking advanced functionality away from technical users: with careful thought and design, both needs can be met with a single standard.  It is also my belief that the most important part of any proposed design is the consideration of users ranging from complete newbies to people that have no time in the world but to learn about every feature that is available and find applications for them.

It is not my belief that mailing list management software should be extended to do things beyond the realm of mailing list management, though I believe that mailing lists are also currently very narrowly defined to be a mirror of public forums such as Usenet, and mailing lists certainly do not need to be restricted to such a model when a little bit of extension can bring a great deal of benefit for all users involved.

In short, mailing list management software should maximize flexibility while minimizing the amount of knowledge required to initially begin using the mailing list system (at the end-user level, at the technical user level, at the administrative level) and regularly interact with the system.  Convenience, ease-of-learning, continued ease-of-use, flexibility in usage patterns/workflows and use cases are all equally important.

Functionality Target Proposal

Let us assume the following statements are true:

1. Mail User Agent software can only be relied upon to have barebones message handling functionality that does not encompass mailing-list aware logic.
2. Users can only be relied upon to have barebones message handling functionality with only message-level context, such that they know that they are responding to a message.

Now, let’s define a few other assumptions that we will take to be true with regard to MUA software:

1. MUA software properly sends either a References header or an In-Reply-To header, which facilitates threading (should not be an issue, since RFC{822,2822,5322} specifies the header and it should be supported universally).
2. MUA software gives the end user no ability to send custom headers as a base feature.
3. MUA software gives the end user no ability to modify headers during message composition other than To, Cc, Bcc, and Subject as a base feature.
4. Even if MUA software can do items 2 and 3 above, it is safe to assume that:
   1. It is terribly inconvenient to do so, or
   2. The user does not know how to do so.

Now, let’s define a few other assumptions that we will take to be true with regard to this ideal mailing list management software:

1. The mailing list management software remembers mailing list threads by remembering message IDs.
2. The mailing list management software may or may not fall back to identification of threads by subject line, timestamp, or some other means, depending on what is most feasible and whether or not there are any accurate huristics that can be applied to make a determination with reasonable confidence.
3. The mailing list management software is remembers messages and posters, and stores information about each message in a manner that makes filtering (e.g., filters for the display of archives, or filters for sending back out to the mailing list) on the server side possible.
4. The mailing list management software can apply a limited number of per-user preferences on the server, including multiple addresses associated with a single subscriber, mode of subscription (digest, normal), interval of digests with at least an option for daily or per-x-number of messages with x being a server-specific configuration item, and perhaps server-side support for bandwidth saving features such as server-side "killfiles".
5. The mailing list management software never requires custom message headers or interfaces other than email for management of the server or per-user preferences, though it can implement such features in addition to a base interface which depends only on an RFC{822,2822,5322} capable mailer.
6. The mailing list management server uses all allowable standard headers for list management such that MUA software implementing those standards can use the information, but does not assume that MUA software is actually compliant to those specifications, instead treating them as optional in nature.

Let’s further define a few modes of operation.  We will discuss potential implementations for these modes later; for now, we are just defining the modes of operation.  If you have a new mode to suggest, please post it in the comments section below, in the same format.  A solution for how to make the mode work is not required, but if desired can be supplied as well if it is in the framework of this post.  We’re talking about the ideal mailing list management software here, not any currently existing one. 

Here is the first (and probably most common) mode of operation, which we will call Mode 1 ("Typical operation"):

1. Bob wants to send an original post to a mailing list (e.g., start a new thread).  He enters To and Subject headers, and composes his message just as if he’s sending to an individual.  Note that current mailing lists already do that, and thus mailing lists are in this respect transparent.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice sees Bob’s message, and wishes to add to the thread of discussion, so she hits "Reply", and enters her message as a reply to Bob’s.  She then hits send.
4. Alice’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.

Here is Mode 2 ("Private replies"):

1. Bob wants to send an original post to a mailing list (e.g., start a new thread).  However, the thread is intended to be a private poll, and so replies should go to Bob directly and not be distributed to the mailing list.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice replies to Bob’s mailing list post, which goes to Bob without being distributed to the mailing list.

Here is Mode 3 ("Private subthreads"):

1. Bob wants to send an original post to a mailing list (e.g., start a new thread) normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message with a few queries and sends the message to the mailing list.
4. Frank responds to Alice’s queries with a few more to seek clarification.
5. Bob responds to Alice normally.
6. Bob responds to Frank normally.
7. Jo sees Bob’s response to Frank and has a question that should be addressed to Bob directly, without going to the mailing list.  Jo sends an email that should get directly to Bob and not be distributed to the mailing list.
   1. Note: mail filtering (if Bob uses it) should not filter the mail as if it came from the mailing list, but the message should thread properly if mail filtering is not considered or if the message is moved to a folder with the rest of the mailing list messages.
   2. Bob receives the response and replies to the message, which goes to Jo and does not get distributed to the whole of the mailing list.

Here is Mode 4 ("Dead thread"):

1. Bob wants to send an original post to a mailing list that is normally a discussion list, but this is an announcement and there should be no replies.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message with some questions pertaining to the announcement.
4. The mailing list software informs Alice that responses were disabled on the post.

Here is Mode 4a ("Dead subthread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message with some questions.
4. Bob responds to Alice’s message, answering the questions, and noting that the discussion should go elsewhere for any reason, and that the subthread is closed.
5. Richard responds to Bob’s message, trolling.
6. The mailing list software informs Richard that responses were disabled on the subthread.

Here is Mode 4b ("Killed thread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message.
4. Frank responds to Bob’s message.
5. Jo responds to Bob’s message.
6. Frank responds to Alice’s message.
7. Frank responds to Jo’s message.
8. …
9. Eventually, the thread gets out of hand, and everyone but the trolls want the thread dead.  Bob sends a message stating that the thread is dead, and dead it remains.
10. Future messages in reply to the thread are responded to by the mailing list software with a message stating that the thread was killed and replies are disabled on the post.

Here is Mode 5 ("Muted thread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Conversation ensues and certain users are utterly disinterested and do not want to receive messages on the thread any longer.
4. Jo sends a message to the mailing list indicating disinterest and the mailing list server obliges by not sending Jo any more messages from the thread.

Here is Mode 5a ("Muted subthread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Conversation occurs.
4. Jo sends a message to the mailing list indicating disinterest in a subthread, and the mailing list server obliges by not sending Jo any more messages from that subthread.

Here is Mode 5b ("Temporary Mute"):

1. A mailing list has "blown up" and some of its users would rather not see any mail from the list for a period of time (say, a day or a week).
2. Users respond to any message in the mailing list and state that they want the list totally muted from their perspective for a specific (but arbitrary) duration.

Here is Mode 6 ("Named Subgroups"):

1. Bob is on a mailing list for a project and has a security issue to discuss.  He sends a message with [security] in the subject.
2. The message is delivered to only people that are in the [security] subgroup; whether the subgroup is open for joining by members of the mailing list or is closed off is up to mailing list server configuration.
3. Replies are contained within the named subgroup.

Here is Mode 7 ("Hidden Addresses"):

1. A group of users would like to have a mailing list, but they do not want their addresses shared with other users when they post, for whatever reason.
   1. Note that this can be used to prevent spammers from subscribing to the list, lurking, and harvesting addresses–very useful today.
2. The mailing list software takes posts and:
   1. Modifies the "From" header of the message to retain the name component, and changes the address component to that of the mailing list.
   2. Replaces the Message-ID header of the message with one generated by the mailing list software for the mailing list itself.
   3. Removes any Received headers.
   4. Removes (but remembers!) any Cc headers that were applied to the incoming message, and honors them until such time as it is told that the Cc should be dropped explicitly, for example by a recipient of the Cc (who sees the Cc line with only their address in it) or a trust list administrator or trusted list helper that can act on the behalf of the Cc recipient ("plz stop sending mailz 2 me k thx").

Possible Implementation (High Level) Notes

Now, between these seven distinct modes and their submodes, all the infrastructure that is necessary to expand the mailing list’s utility are possible to fit nearly any type of usage pattern/use case–transparently, and without any special requirements on MUA software beyond very simple, basic message receiving/display and sending.  At this point, I’m interested in collecting other use cases that I may not have thought about, but that people already use or would imagine would be useful.  The goal, ultimately, is to attempt to come up with a design and specification for a truly transparent mailing list management system that is more flexible than today’s software, that can have per-user preferences that are centralized and MUA-independent, and have features that can be used in the context of messages sent to the mailing list.

One idea is to use in-band signaling on the mailing list to indicate various things.  Unlike list software like majordomo where commands are sent to special mailing addresses, it’s possible that some things can be done by adopting existing conventions and others can be done by adding special information to the message body itself.  For example, Mode 6 is imagined as being part of the already-widely adopted convention that people can filter out based on [tags] in the subject, so an [OT] is used for off-topic, [RFC] for request-for-comments, and so forth.  The advantage of adopting conventions and in-band signaling is that it is inherently portable to nearly all MUA software, regardless of age or capability, as long as either References are In-Reply-To are supported.

Additional Modes

If you know of a workflow or a mode of operation/use case that would not fit in with this proposal, state it (I’m not looking for technical arguments of feasibility right now, just desired modes of operation; a future post and/or wiki page will serve the purpose of overall technical feasibility evalutation).  Consider this to be the "research stage" of a system to-be-specified, with potential for becoming an RFC a published Internet standard with the goal of improving life for all of us: non-technical users, technical users, users married to our workflow (or not).  The only correct system is one that works for everyone in as flexible manner as is possible.

Centralization

Many of these use cases (modes) require that mailing lists are a "gateway" for all of the interaction between users unless things are explicitly taken off-list, which of course users are free to do, if desired.  Some would seem to believe that if everything is centralized that certain use cases become inherently impossible.  While that might have been true given the computing resources that were available in the 1990s, it is not true today, and we shouldn’t pretend that it is; even a uniprocessor server running with a 266 MHz CPU and 64 MB of RAM should be able to run a mail list server implementing these features, serving a moderate number of users and still be fast and responsive.

Sure, lists that are the size of the LKML would probably want to run the mail list server on one machine and the database server on another, or have a distributed environment where the workload can be handed between mail list servers running for the same list (or family of lists), but if the software is correctly designed and implemented with such a configuration in mind, it would be easy for even very large, very active mailing lists to use a featureful system such as this.  For that matter, correctly designed, "centralized" does not have to mean "not highly available", because here I am talking about the service being centralized for a single list, although it would be possible in theory to have the mailing list run on only donated CPU time from volunteers (that said, DNS already makes this possible when combined with a well-planned and well-executed deployment process, though not terribly easily on the IPv4 Internet).

Others state that a centralized setup would take away their freedom to interact with the mailing list in some way: I am saying that this doesn’t have to be the case.  It may be with the way current mailing lists are implemented, but it doesn’t have to be that way at all!  And it can be done in a way that is intuitive–or at the very least, makes sense to users and is easy to learn, if not 100% intuitive.

Again, however, the motivation at this point is a high-level conversation that seeks to be gradually refined into lower and lower levels of abstraction until the final product is source code that can be seen and examined.

Motivation

Why am I motivated now?  Because it is plain as day that a universally applicable solution can be made to exist, and it’s even clearer that there’s no reason for it not to.  I don’t expect that this would be an overnight process, but I think that given ample discussion and interest, we can figure out a way to make everyone happy, even those that are way out with the extreme corner cases.  After all, there is no reason that systems cannot serve both complete end users and extremely adept technical users seeking to use every available feature.

In short, I think it’s time for a change in the way we think about what mailing lists are and how they work, and I think that it’s time that we consider how to make them useful and transparent to us, all of us, regardless of technical knowledge or background, regardless of the end user’s stringent desire to follow RFCs, and without having to be concerned about everything the system can do (while knowing that if you want to learn about all of the features and use them all, you certainly could).  I think it’s time to drop the assumptions that we’ve made for the last several decades with regards to mailing lists and redefine them.

So, who is interested?

Of the many controversies in free software, this is one that I have long found to be interesting.  People seem to define things in different ways, which leads to confusion (and arguing) when one person calls a system a "GNU/Linux" system and the next calls it just the "Linux" system.  Too often, I have found people saying "You must always call it GNU/Linux, Linux is nothing without GNU!" and this is simply incorrect.  So, here I try to spell out the issues and when something is "GNU/anything" and when it is not.

Question 1: What is an operating system?

Here is a (very partial, heavy on the Unix-like systems) list of operating systems:  GNU, FreeBSD, OpenBSD, NetBSD, Microsoft Windows, Mac OS, XENIX, AIX, OS/2, MS-DOS, PC-DOS, FreeDOS.  All of these are full operating systems, though some are more compartmentalized than others.  I hear some saying already, "But GNU is not an operating system!"  Incorrect.  When using the Hurd and a GNU userland operating system stack, that is the GNU system.  I hear others saying, "But you left Linux out!"  Correct, I did: Linux is not an operating system by itself.  Many people are under the impression that the kernel is the operating system, and this is simply incorrect.  A kernel is the core of an operating system, but it is not the operating system itself.

If the kernel is not the operating system itself, what is?  An operating system consists of a kernel and software that is used to make that kernel useful—software that provides the user (or the programmer) with an interface to the kernel.  An operating system in the modern sense comprises a kernel and software that surrounds the kernel.  In the case of POSIX, UNIX and UNIX-like operating systems, this includes the C library, a set of utilities (kill, ps, ls, cp, mv, rm, sh, and so forth), and more.  To compare to Windows, the Windows operating system is not just NTOSKRNL.EXE.  That’s the kernel, but there are a suite of executables and libraries that surround it and provide interfaces to it which make the kernel useful.  While it is technically possible to do, law prevents the creation of a "GNU/Windows" or "GNU/NTOS".  It would not be as straightforward as it is to run GNU on a Unix-like system, but it would be possible (Cygwin is as close as it can lawfully get).

So, when we talk about the FreeBSD operating system, we are referring both to the FreeBSD kernel and the utilities and software that make it useful.  "FreeBSD" includes utilities and libraries and all of that, just like the NetBSD or Windows operating systems do.  But, Linux does not.  When we say "Linux", we are necessarily talking about only a kernel, the project that is the brainchild of a man named Linus Torvalds and the army of programmers around him.

Question 2: What is an OS kernel?

An operating system kernel is the core of the operating system.  However, without the rest of the components that build an operating system, a kernel is useless.  To make matters more interesting, a single kernel can be used as a component in several different operating systems.  This is actually the case for many systems, though it is most prominent in the case of systems that are built around the Linux kernel because there are so many of them.

The lifecycle of a kernel is generally something like this:  Set up and initialize itself, load and initialize basic hardware drivers, kick off one (or more) processes that are required to make the system useful, and then loop, handling things like hardware interrupts and system calls from software.  In the case of a Unix-like system, the kernel attempts to run a program called "init" (usually on modern systems, this is found in /sbin/init), which initializes userland and starts running services that make the system useful.  So a Unix-like operating system provides at least a kernel and an init process, as well as programs for logging in from TTYs (or virtual terminals), manipulating files, processes, and the kernel’s state, and so forth.  If a Unix-like system cannot find an init program, it will halt (panic) or return to firmware where a user could tell the kernel to use an alternate program for init (for example, you can pass "init=/bin/bash" to the Linux kernel, and it will start a single program: bash).

Question 3: Where is the line between an OS and the rest?

To answer this question, we must look at several operating systems, and we must be able to know what the difference is between core software and application software.  The line can be blurred, of course, especially when the core software depends on certain application software.  So in order to more accurately tell the difference, we need to know exactly what the core system uses.  A classic example in the UNIX world: some software is used as system software, and is also available for use as application software.  A system cannot be called a Unix without some of it, and a system certainly isn’t Unix-like without it (and remember: Unix is an operating system definition).

For non-technical people, an operating system is a distribution.  That is, Microsoft Windows is an operating system as it is installed; Ubuntu, Debian, and Red Hat Enterprise Linux are also operating systems.  For those who are more technical, though, the line isn’t there:  the line is somewhere between what is shipped for installation and what is the core system.

Let’s compare FreeBSD with Linux.  A typical FreeBSD system contains the FreeBSD kernel, FreeBSD’s C library, FreeBSD’s kernel management utilities, an implementation of init, and implementations of the Unix utilities (ls, ps, cp, kill, etc.).

However, a typical Linux system contains the Linux kernel, the Linux kernel utilities, the GNU C Library, GNU coreutils, GNU findutils, GNU bash, filesystem-specific utiltiies, an implementation of init (could be sysvinit, Upstart, or any number of other ones), and others.

Question 4: When is it GNU/Linux?  When is it not?  When is any system GNU/anything?

The simple answer: most systems aren’t GNU/anything, but they could be.  Take as an example the project to use the FreeBSD kernel (but not the FreeBSD userland).  The system cannot simply be called FreeBSD any longer: a significant portion of the FreeBSD operating system is removed when its userland is replaced.  The project’s operating system is called "GNU/kFreeBSD", not "GNU/FreeBSD" because FreeBSD was a complete operating system before its utilities were removed and GNU implementations dropped in.  This would include the GNU C library and the typical things you see in a Linux installation.  This also means that it’s a different operating system entirely!  Any piece of software that expects to be built on FreeBSD and assumes a complete FreeBSD system might not even build when using the FreeBSD kernel and the GNU userland.

In the case of Linux:  When you’re using the GNU utilities and core system, it’s GNU/Linux.  When you’re not, then it’s not.  For example, Android is a distinct operating system from Ubuntu.  They both use the Linux kernel, but that’s where the similarities stop.  Android is one example of a distinctly different operating system that shares a kernel with another operating system.  Aside from the fact that Ubuntu is most commonly run on x86 or x86-64 systems and Android is most commonly run on ARM systems, if you sat the two side-by-side on the same platform you’d still have different operating systems that do things differently.  This is because Android doesn’t even use the same C library as Ubuntu; many pieces of software that are stated to work on a Unix-like system such as GNU/Linux or FreeBSD assume a richer environment (closer to that specified by operating system standards).  They would have to be ported to Android to run on Android successfully; that’s a major red flag that says "Oh, that’s a different operating system."

Question 5: Does that mean I can create my own operating system?

Yes.  You can.  And you can even re-use the kernel from any other operating system that is licensed in such as way as to permit you to do so.

Let’s think about this in terms of OS X and what Microsoft could potentially do for Windows, as an example.  Apple used to have its own operating system called "System", which was then called "MacOS", and today we call it "Mac OS Classic".  This system was a cooperatively multitasking operating system that ran software specially built for it.  It was very popular, and it was also not as robust as modern operating systems.  It was pretty interesting, though: it did not have a command-line interface.  Nearly everything that was in the System Folder was the operating system itself, aside from installed fonts, extensions, and system enablers (though often extensions and enablers would patch the operating system in memory, or hook into it in some way so as to attach itself to it).  Apple abandoned that system years ago for OS X.  They took a BSD kernel and wrote a stack around it, reusing some components but using a very different architecture to put things together.  The kernel for OS X is called Darwin, and it’s a very important—in fact, the central—piece of the OS X operating system stack.  But it is not OS X; it is but a component.

How could Microsoft do something like this?  Let’s say that Microsoft went and did something similar, taking the BSD kernel and porting Windows’ software stack to run around it.  They also call it Windows.  Let’s say that they modified the kernel to load Microsoft’s old drivers and that they had incorporated a PE loader and ported their software to this new operating system (but retained API and binary compatibility with their legacy operating system that used a very different composition of kernel and userland, by necessity).  Would this mean that they were running FreeBSD?  No.  It would mean that their operating system were based on (at the very least) the FreeBSD kernel.  Though they could keep the FreeBSD stack and package it as a set of extensions to their new operating system, and claim the ability to run FreeBSD (and in a limited fashion, some Linux, if they retain the Linux compatibility mode of the FreeBSD kernel and system) software.  Now that would be an interesting market position for Microsoft to be in, wouldn’t it?  Microsoft could even make their new system compliant in some way with POSIX and the SUS and get branded as a UNIX® system.  But the new system would be more like Windows than it would be FreeBSD if they didn’t keep FreeBSD as-it-were, since they’d likely replace the core stack to fit their desires and needs, like Apple did.

Conclusion

I hope that this has clarified things a bit.  However, if it has not, a good place to start reading is the Wikipedia articles for "operating system" and "Kernel (computer science)".  They are a long articles, but they make for a great starting point for research and clarification of just what an operating system is.  Both articles look at different operating systems (and their kernels).  As to whether or not you can determine what an operating system is, you will want to become intimately familiar with many different operating systems (and types of operating systems; there are families of them as people create more and more systems both original and reimplementations of others) before you start trying to figure it out.  Look at native code and managed code operating systems—also realize that the distinction of what is the operating system can sometimes be relatively hard to determine accurately.

If you want to see what a bare operating system looks like, though, check out one of these operating systems’ source code:  FreeBSD, NetBSD, OpenBSD.  Each of them in their version control systems has the complete operating system, which is much more than just a kernel.  You can then, should you want, figure out exactly what packages on a typical GNU/Linux system take the place of the remainder of the operating system.  It’d be an interesting exercise in learning—if learning is truly what you’re after, and not just argument.

Richard Stallman.

That’s right.  Now, there are many things that the man has done that have been effective, and the man has worked hard to ensure that we have freedom in the vast world of software.  But recently, he has stated that Miguel de Icaza “is basically a traitor to the free software community,” and no matter how much good a person has done, I cannot stand by such a fallacious statement.

What is freedom, people?  The two cornerstones of freedom are education and choice.  A person who is uneducated cannot choose, and so one depends on the other.  Some people will choose to use either free software or proprietary software exclusively, and others will mix the two together, with varying priorities that they place on their choices.  As long as people are aware of what’s out there and can determine what is best for the way they use their own systems, freedom is available.  Yes, that means some people may choose to run Windows.  Many will not, and if one makes a conscious choice, that is for them to do for themselves.  My concern is that people need to know about the choices in order to make a choice to begin with.

Now, de Icaza has done a great deal of work for the open source community.  He’s one of the key people who has been behind Midnight Commander and GNOME.  And of course, Mono.  He has given us a great deal of good quality, free software that many people can choose to use, and he has given the ability to use a VM-based runtime environment that previously was not cross-platform, as well, on nearly any platform that can be chosen.  The GNU people are even working on a similar one.  But this is not the reason that de Icaza is supposedly a traitor to the community.  The reason?  de Icaza is helping Microsoft out with their new “Open Source Labs”.

Guess what, Mr. Stallman?  The fact that Microsoft is starting to enter the world of free software cannot be considered to be a bad thing.  You did a great deal of work to ensure that free software that is adequately licensed can stay that way, and the turn the copyright system around on those who have sought out to corrupt it.  You have put free software on the radar around the world.  You have done a great many things that others would not have the heart, nor the motivation, to carry out.  It would appear, however, your time is up:  the world can and will move on without you, and I declare that it is time for you to go away.  When you start attacking people in the very world that you helped to create, you become obsolete and no longer serve your purpose.  I dub St. Ignucius of the Church of Emacs to be a heathen who has forgotten his message, forgotten his values, and is now a harmful creature; hardly a saint, more like a devil.  Overtaken with fits of arrogance.  Lately, you have caused a great deal of infighting within the community.  As I am sure you are well aware, infighting serves very little useful purpose but to tear apart communities.  And now you make a bold, arrogant, and false statement that serves only that purpose, but to an extent the likes of which we have never seen from you.

I say that we shun him—and his kind. Things are starting—albeit slowly—to come around to the way we want them, and he would prefer to attack it.  Why?  What logical reason is there to do so?  Because he’s not the controller of it?  Because it grew to be something that even companies that he helped cultivate an irrational, religious-like hatred of, are beginning to see is useful (not only to them, but to all of us)?  What is his new goal, that all software should be GPL’d and if it isn’t it is evil?  We should not tolerate these sorts of counterproductive behaviors.  Stallman has reduced himself to a Schestowitz, a creature who deserves no respect.  A troll.  I suppose in his old age, he has decided that attention is more important than our freedoms.

Richard Matthew Stallman, you are no longer relevant to our community, our world, the world of people who truly believe in and advocate for freedom, that of choice and that of the rights to study, improve, and distribute software.  Thank you for your services, for they were needed to get us where we are today.  However, you have chosen to do us no further good, only ill.  And for that, I say damn you.  Leave the Free Software Foundation and permit it to continue to work for freedom, or take it down and let the GNU project carry on to produce software without all of the utter crap that you have begun to spew of late.  I have no problem with the GNU project.  I love the GNU project.  And I am grateful for the work that you, independently and through the Free Software Foundation, have done so that we can enjoy the freedom to choose a free software system to do our work and our play.  But you are no longer helping; you have become a bully, an old troll.  You do not help free software any more.  Your recent action is a great offense to free software, and very much condemning a road which will lead to more free software.  Go away, troll.

Long live freedom—true freedom—and free software.  And thank you, Miguel de Icaza, for all of the great software that you have given us, as well, for your continued work to make software more useful and portable.  May you never wind up irrelevant and petty as Stallman has wound up being.

This past week, we were all sick, and I was able to do very little in the way of work.  The kid was rather well off, or so I thought, because he wasn’t sick sick, just came down with a bit of a cough.  Well, now, he’s rather sick.  He is complaining that his ear is hurting, and he is throwing up.  Of course, that’s not good.  The good part, though, is that when his stomach is upset, he gets rather attached to the idea of having a bucket handy, so that he can be prepared for whatever comes out.

I was still feeling rather off this weekend, and so I didn’t make it to this year’s Atlanta Linux Fest.  That made me quite sad.  Come hell or high water, I’m going next year, seriously.  Well, unless I’m sick again, of course, but what are the chances of that happening two years in a row?  Maybe I oughtn’t tempt fate by asking that question.  Oh, well, I did.

Also:  the weather here is fscking crazy.  I don’t know what is up with all of this weather, but we’ve had more rain in the past couple of weeks than we have had in a year or so, in some (if not most) areas.  Lots of things are closed, including schools, businesses, government offices, and roads.  And many people aren’t staying in their homes, and given the nature of the regular climate here, probably did not have insurance against flooding, so they probably are out most everything.  I can only imagine what that must be like, and I don’t envy those that are looking at that situation.

As far as other things go, I should be able to start working on things again relatively soon, after things return to something that somewhat resembles normalcy.  We’ll see about that.  For now, though, I need to go sit with Mr. Benjamin and hope that he drinks something soon and falls asleep soon after that…

I’ve made some CSS changes here, to try to make it easier to read when using cramped display space (and likewise, when using a wide-screen monitor).  Also, posts that contain preformatted text that is wider than the display area will now appear with a scroll bar at the bottom.  I could have introduced breakage here, so if so, send me an email and let me know what’s broken (and in what).  Thanks!

AllTray is a rather small project—for the moment, there is a single developer working on it (myself), and there aren’t a terribly great number of users, though the users that it does have are most excellent.  For a long time, the users didn’t have the ability to observe the development processes for AllTray, nor did they have a bug tracker or any point of contact to the author/maintainer other than email.  Things have gotten better there, since AllTray development takes place in the open on Launchpad (though I do have to get better about publishing feature branches that last longer than a day or two so that people can see them as well while they are in progress).

Many projects that are all about being open and transparent provide mechanisms for their users to get in touch with the developers.  Launchpad has become an excellent place to do these things; Launchpad’s Answers and Bugs components are truly excellent ways to get in touch with people that manage software there.  The system is so good that I’d advocate that every project try to move to it.  Launchpad does have its issues, though they are often quickly fixed (and can be fixed even quicker now that it is free software).  But between Bazaar (a truly amazing and flexible distributed version control system) for code hosting and the other components of Launchpad such as the Answers, Bugs, and Translations components, Launchpad provides virtually everything that a project needs to communicate effectively with its users aside from the project’s Web site.  And users seem to naturally be able to reach out using it.

In the nearly one year that I’ve been working with AllTray, I’ve talked—well, written to—several people who have asked questions and acted in their own ways to work to improve the software.  Development on AllTray has been made much easier just because people are willing to speak up.  Everything from the random “thank you” to “hey, are you going to do this?” or “I’d like to see feature x in the new version” or whatever.  It’s truly great.  I can’t say that every project has users quite like AllTray’s, because AllTray is something of a niche application, but AllTray’s users are a great model for other projects.  There has been virtually no negativity from users and everything has been constructive.

To anyone who manages a project that doesn’t use Launchpad, I’d strongly encourage them to do so.  Its users are great, and as far as management of the overall project goes, it’s very easy for project people to communicate back and manage communications in a single spot.  It’s also great that users can ask a question and that the project can do things like go, “oh, hey, this is really a bug,” or vice versa with bugs→questions.  And if you don’t use Launchpad because you don’t want to use Bazaar (I hear that users of git are pretty dedicated to it; I can understand that, but it’s just not for me—it’s too complex, and I like that Bazaar just stays out of my way), consider using LP for everything but code hosting.  Or consider using Launchpad’s code hosting to mirror your project from git, it does this quite nicely.  Or, find some way to do what Launchpad does wherever you do project management.  It works really well, and it’s a big help.

And to AllTray’s users. A big Thank You.  You guys and gals are great.  Keep communicating!

In 40+ years of computing, there is bound to be software and tools that people are simply not aware of, or have forgotten about, or have moved from the forefront to behind the scenes—there are actually lots of these.  One of the reasons that I watch Freshmeat for software releases is to constantly look for things that I might be able to use or are just plain nifty.  After all, there is a lot of software out there.

One piece of software that is almost ubiquitous (at least, outside of the not-so-wonderful world of Microsoft Windows) is a small program known as m4.  You may or may not have heard of it, and you have probably never used it outside of the GNU build system if you’ve used that at all.  m4 is fantastic, if hard to learn, but it is available and it is a pretty much standard component of most UNIX-like systems.  It is a macro processor that dates back to V7 UNIX and it’s been standardized as part of the POSIX body of standards, and GNU has its own version with extensions.

Essentially, what you do with m4 is create macros that process text, and then you can use those macros to simplify things.  In other words, m4 makes it possible to follow the DRY principle when it comes to writing text documents, configuration files, and other text-based files.  For example, I’ve written some macros to make it easier to work with djbdns configuration files, which are available on my software page.  Instead of writing a line that looks like this:

+alltray.trausch.us:173.15.213.185:600

I can write a line that looks like this:

DD_A_FULL(alltray, 173.15.213.185)

Or, for multiple hosts that share the same address, I can (actually, I do; this is from my configuration file) do this:

DD_A_IP(173.15.213.185)
DD_A(allspice)
DD_A(alltray)
DD_A(mike)
DD_A(morganne)
DD_A(phone)
DD_A(projects)
DD_A(sip)
DD_A(vcs)
DD_A(www)
DD_A(gallery)
DD_A(wiki)
DD_AAAA_IP(2001:470:1f11:3f::1)
DD_AAAA(allspice)
DD_AAAA(alltray)
DD_AAAA(mike)
DD_AAAA(morganne)
DD_AAAA(phone)
DD_AAAA(projects)
DD_AAAA(sip)
DD_AAAA(vcs)
DD_AAAA(www)
DD_AAAA(gallery)
DD_AAAA(wiki)
DD_AAAA(spicerack)

The nifty part is that the AAAA lines actually expand to:

:allspice.trausch.us:28:\040\001\004\160\037\021\000\077\000\000\000\000\000\000\000\001:600

Which, as you can imagine, is quite a pain to manage manually.  However, using m4 and some helper programs that are written in C for formatting IPv6 addresses for AAAA records and formatting SRV records for things like XMPP services, managing my configuration has become so easy that I don’t really have to think very hard to do it.  I don’t have to seek out any online generators for SPF records, AAAA records, or SRV records.  I don’t have to worry about repeating a TTL in every line, either, because I set up a domain like so:

DD_DOMAIN(trausch.us)
DD_TTL(600)
DD_SOA(spicerack,mbt.zest.trausch.us,1800,600,21600,600)
DD_NS(173.15.213.185,spicerack)
DD_NS(,primary.staffasap.com)
DD_PTR(173.15.213.185,spicerack)

And for a section that should have a longer TTL, I do so by saying DD_TTL(longer_ttl) and then writing the new lines.  I didn’t try to turn the configuration file format into something that was BIND-like (after all, part of the reason I left BIND was I hated managing its configuration files) but I did make the configuration easier for me to manage.  Now, when I need a new address in the file, or change anything about my domain, it’s as simple as adding or deleting macro calls, and the rest is handled for me.  (I also modified my Makefile that generates the djbdns "data.cdb" file so that m4 is called automatically when I update the data.m4 file.)

M4 can be a major pain to use until you learn it.  And while I do sometimes run into the occasional pitfall with it, I’ve used it for adding preprocessing capability to C# and Vala code to make my life easier, and I’ve used it for various other things in the past outside of the GNU build system.  I absolutely love it.  It is not for everyone—indeed, not everyone has a use for a macro processor—but if ever the need arises, it is well worth your time to learn m4 and use it any time you need a macro processor.  It sure beats having to learn macro processors that are specific to a particular environment, and thanks to Cygwin, you can use GNU m4 on Windows, too, if you happen to be so unlucky as to have to live life with 20% of your CPU cycles (and who knows how much RAM) paid as a tax to fend off viruses and malware. 

And no, m4 isn’t just for programming, nor is it just for programmers.  Anyone who works with a great deal of text is likely to have a good use case for it.  People have been known to avoid using things like PHP and alleviate server-side stress by using a set of macros and generating all of their pages statically (without having to statically manage them all).  This can be good if you have to work with a server that is extremely light on resources and shouldn’t be running things like server-side scripting languages and fetching data from databases.  Instead, you can do all of the database lookups and page formulations ahead of time, meaning that the so-called "slashdot effect" doesn’t ever cause you any trouble on nearly any type of server since it is only serving static pages.  Or writing documents that require things like legal boilerplate and the like; though the days of text processing and typesetting for the masses have generally gone away, sadly.  There are a potentially unlimited number of applications that m4 could be used for—and while m4 is somewhat difficult to learn and can be frustrating at times, it’s standard and it has decades of history, like UNIX in general.

Windows 7 Sins.  It is the name of a campaign and Web site launched by the Free Software Foundation in a misguided (IMHO) attempt to publicize some of the issues in software choice.  Right from the start and without reading a word on the Web site, it says two things very strongly:

1. Windows 7 commits sins.
2. Windows 7 users are sinners.

The site provides a decent amount of information.  But I would imagine that a good portion of its potential readers that are linked to the site will stop reading at the domain name, windows7sins.org, just for the two very clear things it says.  Okay, so we have a very strong tie to science, but does that mean that we should alienate people from the start?  I do not think so.  If you are curious, the alleged “sins” are: poisoning education, invading privacy, monopoly behavior, lock-in, abusing standards, enforcing DRM, and threatening user security.

So, we can hold those points as true: Microsoft has done all of these things, and many more, which make supporting it as an organization something that some of us would shudder to think of, and that many more of us simply would not do if given the choice.  And that is the real key to freedom, is it not?  That we should give out the information in a way so as to reach the widest possible audience, that is.  Not that we use emotional manipulation to compel the audience to switch because then all we are doing is forcing them to feel they need to change.  Instead, users should change (or try) when they are ready—not before they are ready, and not because you offended them and made them feel dirty.  That is exactly the same tactic Microsoft uses on its customers in order to convince them of various things like “you get what you pay for,” or “if you did not pay for the software, it is pirated, it’s not legitimate,” or that free software is a form of “malware,” and so forth.  Why should we stoop down to their level?

Wait, am I saying that it is okay if users choose to use Windows?  Yes, I am.  But if Windows is the only thing that you are aware of, and it is all that you know (and believe me, there are still many such people out there) then it is naturally going to be what you are using.  So they’re a “sinner” by default, and that is offensive.  The only thing that they are guilty of is ignorance, and ignorance has a very simple fix: deliver information, let them learn of these things, and then there is no more ignorance.  For many, it is not even intentional ignorance.  They simply do not know.

I think that this campaign would have been a good thing if it did not seek to offend to make its point.  That’s resorting to a very below-the-belt tactic, and it makes the Free Software Foundation (in this case) come out looking no better than Microsoft itself which is unfortunate.  They have done the wrong thing, for the right reason, and it is constructed to make very little go their way.

So, do not refer users to the Windows 7 Sins Web site when attempting to convince them that they should not choose to upgrade to Windows 7 (or are already running it in one of its pre-release forms).  Instead, offer them information.  Explain things to them, and if they aren’t ready to try something else, whatever.  If they want to try something else, and they wind up going back to running Windows for one reason or another, then by all means, let them.  Of course, find out why they did, first, and maybe you can fix something that will eliminate a problem for others, too.

That is freedom.  It’s not free and it’s not easy.  But just remember that users have the freedom to choose, and that is a bigger freedom than what some of them will ever see or understand in free software.  And if they choose to use something we don’t like, that’s fine; it is, after all, their choice.  Who cares if it is not the choice that you would make?

The new server is now in. The whole trausch.us domain is now running from it, aside from one service (my email MX backup for my desktop machine). And even better, it’s being run in a Linux Container.  The server now has the ability to (very efficiently) stand in for at least 10 servers, nicely and efficiently compartmentalized from each other, and after I add a few firewall rules to finalize the containment between public services and network-local (LAN) services for our home network, the containers will have effective separation between them.  The set of rules that I’ll be dropping in will:

• Bar a container from SSH’ing into its host system,
• Bar a container housing public services from SSH’ing into any of its sibling systems that are not also public (i.e., firewalling against the entire RFC1918 set of networks), and
• Bar a container housing completely private services from communicating with any non-RFC1918 addressed machine.

This way, I can ensure that services that I absolutely want kept private (such as the database servers) are that way without chance of mistake.  In situations where one of the private machines must have communication with the public Internet (for example, the database server needs to be able to replicate over the Internet to its sibling), a whitelist rule to permit exactly just that communication will be dropped in, and all is well and good.

My original plan was going to be to use KVM, and have the choice of running any operating system I wanted.  This didn’t work well with the storage strategy that I wanted to employ, though, without writing a lot of custom scripts to manage it for me.  Xen wouldn’t boot a Dom0 Linux kernel on my hardware (and NetBSD would not install successfully), and so I figured I had two reasonable choices left: FreeBSD jails and Linux Containers.  I installed FreeBSD, at first, thinking back to when I ran it on servers as my first choice of operating system.  The more I was thinking about the jobs that the server would be performing—along with the fact that the server was going to be performing many Linux-specific jobs, at least that is in the plans—I realized that jails under FreeBSD would probably not be the ideal solution.  So I started tinkering with container support on Linux.  So far, I am pleased with it.  It’s very much like FreeBSD jails, in that you can have a separate namespace for processes, mounts, and networking.  I think the mounts part is different from jails, since I don’t think you can mount new filesystems in a BSD jail (or at least, couldn’t at the time I used them, when they were very new; they may now have a tweakable or just support it out-right).  That said, it works nicely.  It can be easily managed with default system configuration files and minor additions to the init scripts.  (It can probably even be managed as an Upstart job, as I do with my DNS server, but I haven’t gotten there just yet.)

Currently, there are only two containers running on the server, though that will change very soon.  Containers/VMs were the very reason I needed to make this setup in the first place…

In Other News…

I am still stuck on AllTray.  I somehow still do not completely understand the code that I’m going to need to write to get the minimize-instead-of-close feature to work properly, nor how to elegantly handle various situations that it seems like will arise once it is written.  Once I “get it” (or someone else is able/willing to make it happen), the next release of AllTray will be made.  It looks as though that will definitely not be in time for inclusion in the forthcoming distribution releases for the latter half of the year, but I will try to build packages for then-current distributions and work to get the new AllTray into the next versions of distributions soon after that.  I’m not giving up; I’m just stuck at the moment.