Frequently enough, I see people forwarding messages to mailing lists that were intended to get to the ML in the first place, but didn’t happen due to some configuration issue or whatever.  There are people that think this is fine, this is workable, and it is correct.  I happen to disagree, but discussing this in the context of an existing mailing list software’s mailing list is proving to be fruitless (I would have guessed this initially anyway, but it wasn’t my idea to take the conversation there…).  So, here, I attempt to state exactly where I’m coming from as a starting point.  This is the first part of what will hopefully be a productive series of posts that strive to define what mailing list software ought to be, how it ought to work, and gradually refining detail until it becomes a blueprint for an initial implementation, at which point it would be my intention to begin a project to do the implementation.

Base Position

Mailing lists should Just Work.  Mailing list management servers are software just like any other software, and they can be made to do what is convenient for us humans, and they should be made to do what is convenient for us humans.  Mailing list software should cater to the needs and desires of all its users, regardless of technical aptitude or choice of MUA (mail user agent, email client) software.  Mailing list software should be updated to operate with the expectations and flexibility expected of software used in the 21st century, and provide more options for interaction both technical users and non-technical users alike can love and use.

It is my personal belief that both mailing list software and MUA software should follow the established philosophy of "do one thing and do it well".  That means that MUA software should be expected to only be capable of being a basic mail user agent, and that mailing list server software should be expected to be capable of everything required to manage a mailing list for end-users, technical users, and list administrators.  A finer-grained permission model in mailing list server software could be adopted to make some of the proposed features even more convenient and get rid of the "administrator" vs. "user" line (i.e., "Jack can unsubscribe users", or "Carla can list messages in the mailing list’s pending approval queue", or whatever).

Furthermore, it is my belief that enabling convenience for end-users does not necessarily mean taking advanced functionality away from technical users: with careful thought and design, both needs can be met with a single standard.  It is also my belief that the most important part of any proposed design is the consideration of users ranging from complete newbies to people that have no time in the world but to learn about every feature that is available and find applications for them.

It is not my belief that mailing list management software should be extended to do things beyond the realm of mailing list management, though I believe that mailing lists are also currently very narrowly defined to be a mirror of public forums such as Usenet, and mailing lists certainly do not need to be restricted to such a model when a little bit of extension can bring a great deal of benefit for all users involved.

In short, mailing list management software should maximize flexibility while minimizing the amount of knowledge required to initially begin using the mailing list system (at the end-user level, at the technical user level, at the administrative level) and regularly interact with the system.  Convenience, ease-of-learning, continued ease-of-use, flexibility in usage patterns/workflows and use cases are all equally important.

Functionality Target Proposal

Let us assume the following statements are true:

1. Mail User Agent software can only be relied upon to have barebones message handling functionality that does not encompass mailing-list aware logic.
2. Users can only be relied upon to have barebones message handling functionality with only message-level context, such that they know that they are responding to a message.

Now, let’s define a few other assumptions that we will take to be true with regard to MUA software:

1. MUA software properly sends either a References header or an In-Reply-To header, which facilitates threading (should not be an issue, since RFC{822,2822,5322} specifies the header and it should be supported universally).
2. MUA software gives the end user no ability to send custom headers as a base feature.
3. MUA software gives the end user no ability to modify headers during message composition other than To, Cc, Bcc, and Subject as a base feature.
4. Even if MUA software can do items 2 and 3 above, it is safe to assume that:
   1. It is terribly inconvenient to do so, or
   2. The user does not know how to do so.

Now, let’s define a few other assumptions that we will take to be true with regard to this ideal mailing list management software:

1. The mailing list management software remembers mailing list threads by remembering message IDs.
2. The mailing list management software may or may not fall back to identification of threads by subject line, timestamp, or some other means, depending on what is most feasible and whether or not there are any accurate huristics that can be applied to make a determination with reasonable confidence.
3. The mailing list management software is remembers messages and posters, and stores information about each message in a manner that makes filtering (e.g., filters for the display of archives, or filters for sending back out to the mailing list) on the server side possible.
4. The mailing list management software can apply a limited number of per-user preferences on the server, including multiple addresses associated with a single subscriber, mode of subscription (digest, normal), interval of digests with at least an option for daily or per-x-number of messages with x being a server-specific configuration item, and perhaps server-side support for bandwidth saving features such as server-side "killfiles".
5. The mailing list management software never requires custom message headers or interfaces other than email for management of the server or per-user preferences, though it can implement such features in addition to a base interface which depends only on an RFC{822,2822,5322} capable mailer.
6. The mailing list management server uses all allowable standard headers for list management such that MUA software implementing those standards can use the information, but does not assume that MUA software is actually compliant to those specifications, instead treating them as optional in nature.

Let’s further define a few modes of operation.  We will discuss potential implementations for these modes later; for now, we are just defining the modes of operation.  If you have a new mode to suggest, please post it in the comments section below, in the same format.  A solution for how to make the mode work is not required, but if desired can be supplied as well if it is in the framework of this post.  We’re talking about the ideal mailing list management software here, not any currently existing one. 

Here is the first (and probably most common) mode of operation, which we will call Mode 1 ("Typical operation"):

1. Bob wants to send an original post to a mailing list (e.g., start a new thread).  He enters To and Subject headers, and composes his message just as if he’s sending to an individual.  Note that current mailing lists already do that, and thus mailing lists are in this respect transparent.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice sees Bob’s message, and wishes to add to the thread of discussion, so she hits "Reply", and enters her message as a reply to Bob’s.  She then hits send.
4. Alice’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.

Here is Mode 2 ("Private replies"):

1. Bob wants to send an original post to a mailing list (e.g., start a new thread).  However, the thread is intended to be a private poll, and so replies should go to Bob directly and not be distributed to the mailing list.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice replies to Bob’s mailing list post, which goes to Bob without being distributed to the mailing list.

Here is Mode 3 ("Private subthreads"):

1. Bob wants to send an original post to a mailing list (e.g., start a new thread) normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message with a few queries and sends the message to the mailing list.
4. Frank responds to Alice’s queries with a few more to seek clarification.
5. Bob responds to Alice normally.
6. Bob responds to Frank normally.
7. Jo sees Bob’s response to Frank and has a question that should be addressed to Bob directly, without going to the mailing list.  Jo sends an email that should get directly to Bob and not be distributed to the mailing list.
   1. Note: mail filtering (if Bob uses it) should not filter the mail as if it came from the mailing list, but the message should thread properly if mail filtering is not considered or if the message is moved to a folder with the rest of the mailing list messages.
   2. Bob receives the response and replies to the message, which goes to Jo and does not get distributed to the whole of the mailing list.

Here is Mode 4 ("Dead thread"):

1. Bob wants to send an original post to a mailing list that is normally a discussion list, but this is an announcement and there should be no replies.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message with some questions pertaining to the announcement.
4. The mailing list software informs Alice that responses were disabled on the post.

Here is Mode 4a ("Dead subthread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message with some questions.
4. Bob responds to Alice’s message, answering the questions, and noting that the discussion should go elsewhere for any reason, and that the subthread is closed.
5. Richard responds to Bob’s message, trolling.
6. The mailing list software informs Richard that responses were disabled on the subthread.

Here is Mode 4b ("Killed thread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Alice responds to Bob’s message.
4. Frank responds to Bob’s message.
5. Jo responds to Bob’s message.
6. Frank responds to Alice’s message.
7. Frank responds to Jo’s message.
8. …
9. Eventually, the thread gets out of hand, and everyone but the trolls want the thread dead.  Bob sends a message stating that the thread is dead, and dead it remains.
10. Future messages in reply to the thread are responded to by the mailing list software with a message stating that the thread was killed and replies are disabled on the post.

Here is Mode 5 ("Muted thread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Conversation ensues and certain users are utterly disinterested and do not want to receive messages on the thread any longer.
4. Jo sends a message to the mailing list indicating disinterest and the mailing list server obliges by not sending Jo any more messages from the thread.

Here is Mode 5a ("Muted subthread"):

1. Bob wants to send an original post to a mailing list normally.
2. Bob’s message hits the mailing list server and is distributed to the subscribers of the mailing list as expected.
3. Conversation occurs.
4. Jo sends a message to the mailing list indicating disinterest in a subthread, and the mailing list server obliges by not sending Jo any more messages from that subthread.

Here is Mode 5b ("Temporary Mute"):

1. A mailing list has "blown up" and some of its users would rather not see any mail from the list for a period of time (say, a day or a week).
2. Users respond to any message in the mailing list and state that they want the list totally muted from their perspective for a specific (but arbitrary) duration.

Here is Mode 6 ("Named Subgroups"):

1. Bob is on a mailing list for a project and has a security issue to discuss.  He sends a message with [security] in the subject.
2. The message is delivered to only people that are in the [security] subgroup; whether the subgroup is open for joining by members of the mailing list or is closed off is up to mailing list server configuration.
3. Replies are contained within the named subgroup.

Here is Mode 7 ("Hidden Addresses"):

1. A group of users would like to have a mailing list, but they do not want their addresses shared with other users when they post, for whatever reason.
   1. Note that this can be used to prevent spammers from subscribing to the list, lurking, and harvesting addresses–very useful today.
2. The mailing list software takes posts and:
   1. Modifies the "From" header of the message to retain the name component, and changes the address component to that of the mailing list.
   2. Replaces the Message-ID header of the message with one generated by the mailing list software for the mailing list itself.
   3. Removes any Received headers.
   4. Removes (but remembers!) any Cc headers that were applied to the incoming message, and honors them until such time as it is told that the Cc should be dropped explicitly, for example by a recipient of the Cc (who sees the Cc line with only their address in it) or a trust list administrator or trusted list helper that can act on the behalf of the Cc recipient ("plz stop sending mailz 2 me k thx").

Possible Implementation (High Level) Notes

Now, between these seven distinct modes and their submodes, all the infrastructure that is necessary to expand the mailing list’s utility are possible to fit nearly any type of usage pattern/use case–transparently, and without any special requirements on MUA software beyond very simple, basic message receiving/display and sending.  At this point, I’m interested in collecting other use cases that I may not have thought about, but that people already use or would imagine would be useful.  The goal, ultimately, is to attempt to come up with a design and specification for a truly transparent mailing list management system that is more flexible than today’s software, that can have per-user preferences that are centralized and MUA-independent, and have features that can be used in the context of messages sent to the mailing list.

One idea is to use in-band signaling on the mailing list to indicate various things.  Unlike list software like majordomo where commands are sent to special mailing addresses, it’s possible that some things can be done by adopting existing conventions and others can be done by adding special information to the message body itself.  For example, Mode 6 is imagined as being part of the already-widely adopted convention that people can filter out based on [tags] in the subject, so an [OT] is used for off-topic, [RFC] for request-for-comments, and so forth.  The advantage of adopting conventions and in-band signaling is that it is inherently portable to nearly all MUA software, regardless of age or capability, as long as either References are In-Reply-To are supported.

Additional Modes

If you know of a workflow or a mode of operation/use case that would not fit in with this proposal, state it (I’m not looking for technical arguments of feasibility right now, just desired modes of operation; a future post and/or wiki page will serve the purpose of overall technical feasibility evalutation).  Consider this to be the "research stage" of a system to-be-specified, with potential for becoming an RFC a published Internet standard with the goal of improving life for all of us: non-technical users, technical users, users married to our workflow (or not).  The only correct system is one that works for everyone in as flexible manner as is possible.

Centralization

Many of these use cases (modes) require that mailing lists are a "gateway" for all of the interaction between users unless things are explicitly taken off-list, which of course users are free to do, if desired.  Some would seem to believe that if everything is centralized that certain use cases become inherently impossible.  While that might have been true given the computing resources that were available in the 1990s, it is not true today, and we shouldn’t pretend that it is; even a uniprocessor server running with a 266 MHz CPU and 64 MB of RAM should be able to run a mail list server implementing these features, serving a moderate number of users and still be fast and responsive.

Sure, lists that are the size of the LKML would probably want to run the mail list server on one machine and the database server on another, or have a distributed environment where the workload can be handed between mail list servers running for the same list (or family of lists), but if the software is correctly designed and implemented with such a configuration in mind, it would be easy for even very large, very active mailing lists to use a featureful system such as this.  For that matter, correctly designed, "centralized" does not have to mean "not highly available", because here I am talking about the service being centralized for a single list, although it would be possible in theory to have the mailing list run on only donated CPU time from volunteers (that said, DNS already makes this possible when combined with a well-planned and well-executed deployment process, though not terribly easily on the IPv4 Internet).

Others state that a centralized setup would take away their freedom to interact with the mailing list in some way: I am saying that this doesn’t have to be the case.  It may be with the way current mailing lists are implemented, but it doesn’t have to be that way at all!  And it can be done in a way that is intuitive–or at the very least, makes sense to users and is easy to learn, if not 100% intuitive.

Again, however, the motivation at this point is a high-level conversation that seeks to be gradually refined into lower and lower levels of abstraction until the final product is source code that can be seen and examined.

Motivation

Why am I motivated now?  Because it is plain as day that a universally applicable solution can be made to exist, and it’s even clearer that there’s no reason for it not to.  I don’t expect that this would be an overnight process, but I think that given ample discussion and interest, we can figure out a way to make everyone happy, even those that are way out with the extreme corner cases.  After all, there is no reason that systems cannot serve both complete end users and extremely adept technical users seeking to use every available feature.

In short, I think it’s time for a change in the way we think about what mailing lists are and how they work, and I think that it’s time that we consider how to make them useful and transparent to us, all of us, regardless of technical knowledge or background, regardless of the end user’s stringent desire to follow RFCs, and without having to be concerned about everything the system can do (while knowing that if you want to learn about all of the features and use them all, you certainly could).  I think it’s time to drop the assumptions that we’ve made for the last several decades with regards to mailing lists and redefine them.

So, who is interested?