What is NAT?

NAT, or network address translation, is a mechanism that attempts (and fails, in many cases) to provide transparent access to the Internet for multiple IP-networked devices that can not all have public IPv4 addresses. For example, it is used in homes and many small businesses to provide Internet connectivity to multiple computers on a single connection to the Internet with a single public IP address. NAT was invented in the early 1990s (approximately 1993, if memory serves) in an attempt to delay the exhaustion of the IPv4 address space. It was effective, too, for that purpose—we probably would have ran out of IP addresses ten years ago had NAT not come into existence.

NAT has one major advantage: it enables an entire network to share a single IP address, thus conserving address space. However, NAT comes with a number of disadvantages at different levels. Some of these disadvantages are:

• Increase in network operations overhead. Most types of NAT require the ability to maintain additional tables in memory which support the task of both address and port number translation.
• Having a NAT increases the complexity of a network at the IP layer. This will be discussed in more detail in a few minutes.
• Any NAT device will perform more slowly and consume more resources than a plain router—for most computer systems, this is not a major problem. However, for networks with high-bandwidth Internet connections which are using embedded devices for routing (such as a simple, consumer-grade wireless router) this can be a problem. It can also be a problem for any network that has very old routing equipment that has been retrofitted with the ability to perform NAT, as such devices were not designed with enough processor to handle the additional overhead when the network is at full load.
• In network operating systems which behave as routers, NAT is often implemented in the same area as the firewall. For example, in the Linux kernel, the iptables command is used to set up NAT networking on an IPv4 network, and iptables is the front-end to the Linux kernel’s built-in firewall capabilities. This increases the complexity of the firewall code itself, and can make it more difficult to maintain in general, as well as more difficult to audit for security problems.
• End-to-end communication is broken. The Internet was originally designed with the concept of end-to-end communication; that is, one system on the Internet can converse with another system on the Internet directly. Not only does this simplify network design, but it simplifies the design of network applications, as well (particularly those that require bidirectional communication but do not always maintain a persistent connection and require the ability for either side to reconnect upon some sort of a trigger). Some protocols (such as SIP) have worked around this, but such workarounds can be high-cost as well as brittle.

The removal of NAT from protocol stacks therefore yields a number of benefits, including removing all of the disadvantages above. With the transition to IPv6, NAT devices are no longer necessary. Their additional complexity can go away, and networks all around the world will operate more efficiently and with less latency than they do now. It still takes I/O and processor resources to perform normal routing, but nowhere near as much as it does to perform NAT.

Over the years, multiple types of NAT implementations have been created. I am not going to go into a terribly detailed analysis of them all, but they are:

• Full-cone NAT (or, 1:1 NAT). This type of NAT provides no conservation of the address space; one external IP address maps exactly to one internal IP address. While there are uses for this type of NAT, I can think of no use for it that is not better served by another type of device, such as a load balancer, monitoring and failover, or plain routing.
• Address-restricted cone NAT. This is one of the two most common types of NAT. When a system on the inside (usually using RFC 1918 IP address space) sends an IP packet to the outside, the NAT remembers the IP address, protocol, and port of the internal system and relays it to its destination. The destination system may reply by sending packets from any of its own ports to the NAT on the source port that it sent the original packet from.
• Port-restricted cone NAT. This is the other of the two most common types of NAT. When a system on the inside sends an IP packet to the outside, the NAT remembers the same things as for address-restricted cone NAT, but replies from the destination must come from the same port as the packet was sent out to.
• Symmetric NAT. This type of NAT is similar to port-restricted cone NAT.

It is important to consider that a single NAT implementation may combine behaviors from one or more of these types, and some implementations are extremely configurable in terms of what method or methods are used to perform the functions of NAT. It is also important to realize that NAT breaks many legal network behaviors, depending on the application and the type of NAT in use. Various workarounds have been developed in order to traverse NAT devices for some protocols, and sometimes protocols will change in order to add NAT traversal as a core feature, but the overall effect is that NAT (often significantly) reduces network efficiency.

What is NAT not?

NAT is not a security mechanism.

Let me repeat that: NAT is not a security mechanism.

One more time: NAT IS NOT A SECURITY MECHANISM.

I am uncertain where people have gotten the idea that somehow NAT was designed to increase security. It was not. It was designed to help conserve the increasingly scarce resource of IPv4 address space, nothing more. It is not a security tool, and it does not provide any additional security over a properly maintained IP firewall—using a firewall is essential with or without a NAT in place if you need to do any sort of packet filtering at all.

The idea that NAT is a security mechanism probably came from the notion that one cannot see the addresses on the inside of a NAT. However, there are many mechanisms by which a sufficiently interested attacker would be able to determine things such as an approximate (or even an accurate!) count of how many devices are behind the NAT and what their IP addresses are through the use of various protocols, application tricks, and security exploits. For that matter, it is trivial to do things such as setup IPv6 on a NAT’d network and give all the systems on the NAT’d network a globally reachable IPv6 address, all without the cooperation of the NAT device. Only a firewall can stop such a thing, and only if you know what it is that you are trying to stop. And there are some things that even a firewall cannot protect you from (such as trojan horses and intentionally malicious employees and ex-employees). Security is an insanely complex problem to solve, and NAT is not a tool in a security professional’s toolbox.

How does NAT work?

A NAT device has a fully functional IP stack, and operates at a combination of OSI layers 3 (Network Layer) and 4 (Transport Layer)—mostly layer 4. In comparison, a router is an OSI layer 3 device. (In case you haven’t memorized the OSI model yet, refer back to my previous post which shows it.) Let’s say that you have a computer system that is on an IPv4 network, and that IPv4 network is using NAT. When you came to my blog to pull up this post, your Web browser performed the following tasks:

1. It looked in its DNS cache to see if the hostname mike.trausch.us was there. If it was, it used the IP address from the cache; if not, it asked your computer to find the IP address for mike.trausch.us.
2. Then, it asked the operating system to open a TCP socket connected to the IP address for mike.trausch.us on port 80. Since you are reading this, it is probably safe to say that it succeeded, and it was given a socket to work with.
3. It then asked my Web server for the post, which my Web server kindly gave to you. There are a few back-and-forths that I am omitting here for the sake of clarity.
4. The connection from your computer to my Web server was then closed.

In a normal (that is, non-NAT’d) network, this was all nice and direct, and the edge router on your network made it possible for your packets to get here. Even better, in such an event, the chatter in such an event is directly between your computer and my Web server. However, on a NAT’d network, this is not the case. Instead, this is (some of) what happens:

1. Your browser looked in its DNS cache to see if the hostname mike.trausch.us was there. If it was, it used the IP address from the cache; if not, it asked your computer to find the IP address formike.trausch.us.
2. Then, it asked the operating system to open a TCP socket connected to the IP address for mike.trausch.us on port 80.
3. Your computer’s operating system opened the socket, but not to my Web server. It just thinks it did. When the packet went out that was supposed to start the TCP handshake, it ended up at your NAT.
4. Your NAT sees the packet, and makes a note of what the source IP address (your private IP) and source port was.
5. The NAT notes the (source IP, source port) pair, and notes the destination address (e.g., the IP address for mike.trausch.us in this instance), the protocol (in this case, TCP) and sometimes also the port number (in this case, TCP port 80).
6. It then forwards the packet to my Web server on port 80.
7. My Web server receives the packet, which at this point appears to come from your external IP address, and probably a different port from the source port on your computer.
8. My Web server sends a return packet, directed at your NAT device’s IP address and the port that it sent your packet from.
9. When your NAT receives the packet, it looks in its table of entries to see if it has a mapping for the IP address and port it received a packet on.
10. It then forwards my acknowledgement to your computer.
11. All of the rest of the steps are the same, but with the NAT intercepting, looking up, and rewriting every single packet before it is shipped to its destination (either your computer or my Web server).

It is a lot more complex to do all of this, obviously.

And for larger networks, it won’t scale at all: multiple external addresses will have to be used to represent the whole network, because each NAT device can only have a mapping for one system and one (IP, port) combination at a time. What that means is that for larger NAT’d networks, you might have a different “public” IP address for every connection—or if you have a large network and only one external IP address to NAT with, you might actually wind up sometimes not being able to connect to anything at all, because the mapping table in the NAT is full.

So I Have to Learn to Use a Firewall?

Yes. Well, no. Well, sort of.

Consumer devices for use in home networks that support native IPv6 will most likely be running their own firewall with a reasonably sane default set of rules (and hopefully, the ability to change those rules!). For example, not allowing inbound packets to protected ports (those below 1024) and ports well-known to be open and accessible by default by operating systems in the Microsoft Windows family. Of course, it will also be up to people to not install services on their computer systems that are configured to service the world. That’s not terribly hard, given that we have the loopback network (127/8) that is reserved for use locally (such IP addresses aren’t even allowed to reach the physical network outside of a sole system).  This means that a system can run services that aren’t to be exposed to the world (for testing, or in order to protect them from access without first using an SSH tunnel, or for any manner of other things).

Very small, client-system-only networks will most likely use consumer-grade devices, as well, and need not worry about it for the same reasons that home networks won’t likely need to worry about it.

Power users, network administrators, and everyone else in between can instead just configure a firewall. Whatever device is providing core routing functionality more likely than not has the ability to perform firewalling as well—and if not, it’s easy to obtain an operating system and a computer that can perform the task. After all, Linux, the BSD family, and most other UNIX and UNIX-like operating systems not only can function as routers, but can firewall (and Linux and the BSDs are free). If you are the administrator of a network that has more than ten nodes on it, or the administrator of an any-sized network that has more than zero server systems on it, you should know how to use a firewall both in general and the particular implementation that you have.

What About RFC 4193 (IPv6 Private Address Space)?

RFC 4193 does indeed provide for private address space in IPv6. That does not necessarily mean that it has to have NAT. Private address space can be used to ensure that one or more subnetworks have absolutely zero Internet connectivity (or can use something such as an IPv6-enabled SOCKS server in order to have strictly controlled connectivity). However, more often than not, servers that require such security need not use address space reserved for it by RFC 4193, as it would unnecessarily complicate the network. Instead, one could use a single subnet out of their allocation of subnets, treat that subnet as “dark” or private, and configure the firewall to prohibit all (direct) communication with that subnet. If you have a correctly configured network this is trivial.

Private address space in IPv6 can also be useful to create disconnected islands, or testbed networks. However, in production networks, I would expect to see a business that has a /48, for example, simply devote a single subnetwork to private-use.

There are other means by which one can protect their network without NAT; see RFC 4864 (“Local Network Protection for IPv6″) for more information.

So, no NAT in the future?

That’s right. We are heading to a world without NAT, a world where no NAT is needed, and a world where the overhead of the Internet as a whole will be reduced as a result. That pretty much wraps it up for today’s post. Questions? Comments? You know what to do with ’em!

I am sure that there are people who do this in every single line of work. But what disgusts me more than anything is someone who thinks that just because they did something like served in the military or got a degree of some sort that they know everything and that everyone else has to prove that they have two brain cells to rub together. Do you know what someone with a Ph.D. is good at? Marketing. They have to be.

You know what I hate with a passion? Marketing. Because most of the time it is hogwash.

What prompts this, you might wonder?

So today, I responded to an unexpected downtime call. Long story short (because details cannot be given out, for obvious reasons), this meant that I got up out of bed to answer it. No problem; it’s what I do. I go, I handle the problem, I encounter a couple of snags, find that I no longer have authorization to fix those snags, and go on about my day, providing a notification of the issues that I wasn’t able to fix—and why. As far as the “fix”, I had to switch to a backup connection on a mixed voice/data T1 (technically, DS1, but nobody calls it that except the engineers, I am pretty sure). No big deal; the equipment is setup to handle that, and it works just fine, albeit slower than anyone would like it to.

Enter Fatuous. Fatuous is someone who is employed as an “Information Technology” support person. Of course, that is not its real name, but it is suitable nonetheless.

Fatuous sends an email to the effect of “you cannot run data through the T1 because it will make voice calls suffer”. Let’s keep in mind here that this particular equipment does both voice and data, and it gives preference to voice calls. In other words, if all the circuits are busy handling voice calls, there is no more room for bandwidth. Sounds simple, right? It should, because it is.

So, I explain this little fact, and I get a mail back—oh, yeah, and half the office is needlessly carbon copied on this. Great! Let us fill everyone’s inboxes with a bunch of technical jargon that they will not care to read and (probably) have no desire to understand. Well, whatever. So the mail basically says, “Cite something, you’re wrong.” Wait a minute, what? Fatuous seriously does not understand what a T1 is. Now, if you have worked in the IT industry for any period of time, even if you have never used one, you should really know what a T1 is. Especially if you are over the age of, say, 25 or 30. Particularly if you are over the age of 40, since it is quite likely that’s what they were using for high-speed interlinks then (nevermind the fact that it is only a little bit faster than a low-end ARCNET card/network it was once considered high-speed connectivity, just as ARCNET was).

Fine, so I explain what this means in a high-level, cursory overview that hopefully had words small enough for Fatuous to understand. In the meantime, I am raving mad. I have dealt with Fatuous enough that it is readily apparent that there is no salvation here: its ignorance is willfully incurable, and that is terribly sad. I am not sure what is worse: the fact that it has a job doing something I am way overqualified for, or the fact that it has a job doing something that I am way overqualified for and makes a fuck of a lot more money than I do.

Sometimes, I really hate the universe.

First point: FatELF would be useless because “you can do that already, write a cc frontend that compiles the same file multiple times, it’s _not_ hard, I’ve done it before“. Okay, so the proposed solution here is to write a compiler driver that will interpret arguments and, from a single Makefile, build for multiple platforms. There might even be something out there for that, but simply put, if GCC supported this feature intrinsically, then everyone would have it and it would be done in a standard way. Free software works better when everyone can agree on a single standard way of doing things, and not just a single standard template for how it might be done. Using addons to perform this function still yields multiple binaries that have to be shipped anyway, which is decidedly not the aim.

Second point: “how is shipping one (fat) binary ‘better’ than shipping one auto-extracting auto-deciding archive?” Making the assumption that the toolchain and kernel all support the feature as a standard thing here, the difference is simple: the kernel ELF loader would be able to decide which sections of the ELF file should actually be loaded in memory, read only those sections, and go on about its business normally—the rest of the process would not need to change in any way. No temporary copies need to be made, no images need to be extracted, nothing like that has to be done. However, the inverse is quite a different story. Let’s make the assumption that you’re using a POSIX shell script, with the archive of all of the possible binaries appended to the POSIX shell script. First, the script has to be prepended to EVERY such archive (meaning that different versions of the script could exist, and as any programmer knows, DRY), and the script is not going to be trivial: it would have to have code to detect and support every single individual platform. Furthermore, it would require that the user have permission to extract the payload, make it executable, and run it. This is the same deficiency that makes gzexe impractical for everyday use; I know that at least on all the servers that I manage, /tmp is mounted read-write but with execution of scripts and binaries disabled. Finally, it would fail to properly work in the event that something needed to be setuid—that information would have to be in the payload itself, which is absolutely not portable from one system to another. It just cannot be made to work in a generic enough fashion to be reliable on all different types of platforms with different administrative decisions made in the management of those platforms, and in many cases would require an increased attack surface just to be made workable.

However, if FatELF (or, honestly, anything that is truly equivalent) were used, an administrator could copy the binary from one system (say, an x86) to another system (say, a PowerPC) that has all of the other dependencies filled for it, drop it on the filesystem, chown/chmod it once, and it would Just Work. setuid, if needed, would be honored by the kernel, and no extraction has to take place. No additional temporary disk space would be required, nor would it be necessary to incorporate any logic into the ad-hoc “loader” (if it could even be called that) to try to find a filesystem that is read-write with execution permitted for the current user, and therefore no special privileges from the user would be necessary.

In fact, the only way to solve the problem reliably at present would be to have something like /var/cache/adhoc-fat-binaries, and have all ad-hoc “fat binaries” be setuid 0 (or setuid to some user that has all necessary privileges to make something setuid 0 if necessary, probably only UID 0 has that privilege on most systems) so that it could (a) write to /var/cache/adhoc-fat-binaries and (b) set the setuid or setgid bits if necessary for the program to fulfill its function. And it deserves to be restated: we all know that having a single specific standard and adhering to it—even when the standard is less than ideal (and in some cases, like X11, falls quite short of ideal)—is far better than having 100 different and incompatible ways to do the same thing. It’s one of the things that we people in free software know pretty damn well.

See, I don’t see something like FatELF being used for distribution binaries, or anything that would be distributed in an operating system distribution package, except perhaps in special situations where something like biarch is natively supported on the hardware and it would be feasible to permit that sort of flexibility. Instead, I see something like my current situation: I administer several machines for small businesses, and not all of them are the same hardware platform.  They are all the same operating system and many of them have the same libraries installed.  Some of them are 64-bit and some are 32-bit.  Some are x86, some x86-64, and some are neither. But I would very much like to write a single program, say “make” and copy the file to every machine so that it just works. For the moment, if I want something like that, I have to just use something like Java, C#, or a script. Or, if I need something setuid, I do it in C and compile it for every system, shipping the source code file to the systems instead. But it would be more efficient to not have to do that. That is why I would see FatELF being a “good thing”.

I know that I am in the minority.

That brings me to point three: “because in 99% of all usage, the kernel won’t _need_ it. And its cost in effort and overhead would be higher.” For this next part of my post here, I am going to be looking at the Linux kernel, version 2.6.34, which I have just downloaded from kernel.org, which is 64 MB compressed (using bzip2!) and takes up 442 MB when uncompressed, before touching any file in the tree. Now, I am looking at this for x86-64 because that is the system I am running on and typed “make menuconfig”.

Who needs any of the following options? I am willing to bet that the following options are not needed in 99% of all (desktop, server, and embedded, combined) usage:

1. Processor type and features/Support for extended (non-PC) x86 platforms
2. Processor type and features/Maximum number of CPUs
3. Processor type and features/Memory model
4. Processor type and features/Build a relocatable kernel
5. Executable file formats / Emulations/Kernel support for ELF binaries
6. Executable file formats / Emulations/Kernel support for MISC binaries
7. Executable file formats / Emulations/IA32 Emulation
8. Executable file formats / Emulations/IA32 Emulation/IA32 a.out support
9. Networking support/Plan 9 Resource Sharing Support (9P2000) (Experimental)
10. File systems/Second extended fs support
11. File systems/Reiserfs support
12. File systems/JFS filesystem support
13. File systems/XFS filesystem support
14. File systems/GFS2 file system support
15. File systems/OCFS2 file system support
16. File systems/Dnotify support
17. File systems/Kernel automounter support
18. File systems/Kernel automounter version 4 support (also supports v3)
19. File systems/FUSE (Filesystem in Userspace) support
20. File systems/FUSE (Filesystem in Userspace) support/Character device in Userpace support

I can’t even go on. Twenty is enough; I think I have made my point. In 99%+ of all situations, these options are either always on or always off. They are rarely modified. And the kernel still supports a.out from IA32′s really old days‽ Seriously?

What does this tell me? It tells me that FatELF—or anything else that came along and did something like what FatELF would do—has room in the kernel. And if it were for whatever reason incompatible with current ELF (as it would very likely be) then the kernel could still support “old” ELF, without any of the extra fields or sections.

And actually, there is a great deal of possibility around something entirely different altogether. FatELF isn’t the most technically elegant thing I can think of to solve the problems that it solves, but I have yet to see something else seriously proposed. I can think of something even better, actually. We are all taught that operating systems are here to abstract us from hardware, so that we can write applications and not have to worry about communicating with the hardware directly because the OS handles those details for us. Well, if that is the case, then why don’t operating systems also abstract the system’s processor? Why don’t we have operating system kernels that provide a virtual instruction set? Yes, I am talking about essentially moving the application VM into an operating system kernel, though ideally with some supporting utilities in userspace to do things like hold persistent JIT caches and so forth. However, that’s for another post, another time.

We pay cash at restaurants especially ones of certain
nationalities.

The context of this quote is a discussion on credit/debit card usage, and this statement came at the tail end of how care must be taken to ensure that one is not subjected to fraudulent charges (nevermind the fact that banks in the U.S. mostly do zero-liability these days). This has spawned a rather heated discussion, which apparently resulted in the person who made that comment leaving the mailing list.  The whole idea that we continue to change our behavior depending on the ethnicity of the person(s) we are around is nothing short of infuriating. It is like we fail to understand that qualities like honesty and trustworthiness are markers of an individual.

What does the above statement say? It says that the poster of that statement feels that he cannot trust “certain nationalities”. This person later made the claim that it was sad that people could find racism in nearly any remark—but the thing is, it is right there, not even hidden from view. It’s blatant.

Even worse, when the poster of the comment above was called out on it, most of the people on the mailing list jumped to this person’s aid to defend them. Honestly, I do not know what is worse: the fact that this person said this in the first place, or the fact that the majority of the mailing list’s members rallied up on the side of that person. Quite possibly, I think the latter, because that shows that we still have in society the notion that racism is somehow acceptable, and that the more veiled or subtle it is, the more okay and polite it is. I find that downright offensive.

It is certainly enough to make me consider dropping the mailing list and participation in the group altogether. It is difficult to be in a group when you cannot even look at its members—your peers—with respect, even if they are some of the smartest minds you know in a particular field. Why be part of it at all?

So, this is a pretty simple-sounding network, yes? It should be—it is just a handful of computer systems. However, there is a problem—a pretty large one, I think. There is very little in the way of either policy or convention on the network. Some users use their documents directory for storing their documents, others store their documents on their desktop, others somewhere differently altogether on the C drive, others on a network share in a public storage space. There are different versions of software on the various machines, more-or-less updated when someone thinks about it, I think.

This is a perfect scenario which shows why a business network should be centrally managed in some form. Note that I am not saying that each machine should be a bit-for-bit mirror image of the one next to it, though that is certainly a possibility. I think that people should be able to use their own choice of things like email client or Web browser software, because everyone is different. But when you have different client applications fulfilling a role on the individual workstations, you have to take a centralized approach to ensuring that things like the email is all backed up.

Furthermore, if you don’t take a centralized approach to backing up such data, it is very difficult to centralize the network storage. Think about adding a domain controller (that is Windows speak for a central server which handles authentication and authorization, as well as file and printer sharing and things like roaming profiles) to such a network. I expect that with multi-gigabyte mail files, things will be very slow at first—and that likely the only fix for them that is going to be viable in the long term is to centralize more infrastructure.

I am too tired to expand more on my thoughts on what I have learned and where it is heading, but the Reader’s Digest version of the point reads something like this: If you are a small to medium sized business, make sure that you have someone who is competent in both system and network administration, and make sure that they are a part of your business from day one. Like writing software, building up a technical infrastructure without careful thought and design is hazardous and comes with many hidden and unpredictable costs. It is best to head those things off right from the start; to delay only amplifies the cost of fixing the underlying problems and puts oneself in the position where fixing one issue can have a domino-like effect and create more new problems.

For my current situation, I think I am going to have to seriously re-think how this whole setup is done. What I do not yet know is how to quickly and efficiently bring things into shape. A bit of training and education may be required, and certainly the removal of a lot of unnecessarily-granted privileges on the workstations. That, too, should be something caught early-on: do not let every person in a business run with administrator privilege, unless they are an administrator (and even they should only run with administrator privilege when they are actually doing something that requires that privilege). If everyone is an administrator, there is little to no control on how things are done in a network, and it can get messy.

I have a lot more reading to do, as well.

Well, anyway, it is way past my bedtime.  Time for sleep.

The Free Software Foundation objected to UDI for various reasons. Mostly, I think, it was because they were afraid that people who are not them would choose to use drivers that were non-free. As I’ve written about before here, there are people who think that forcing people to use free software is somehow freedom—and I will not go into it in any great depth here, because I have done that in the past. Suffice it to say that forcing anything is not freedom; it cannot be freedom. So, the Free Software Foundation, I think, was really afraid that they would have to do more work to be able to stick to their own requirement of using 100% free software on their own computer systems. (And hey, Roy, if you’re reading—I’m not saying that the FSF is wrong, and I’m not putting myself in a position opposite of that of the FSF. I suspect you think so anyway, but hey, I just figured I would point that out.)

Even if the free software operating systems did not adopt the UDI specification, why didn’t proprietary operating systems? This is perhaps the most puzzling thing to me. It seems that in this event, none of the operating systems—free or proprietary—did what would have made sense. After all, even if only Apple and Microsoft adopted a common device driver specification, that would save a lot of time, effort, and improve user experience all the way around. Apple users would be able to use all the hardware that Microsoft users could use—and the inverse would also be true. The amount of time that device driver authors would have to spend writing and debugging driver code would go way down—free software driver authors would be able to write a driver once, for example, and all systems (including free software systems that chose to support the specification) would benefit.

I could see an objection of a driver specification that was binary-only. However, UDI was not—it mandated an ABI so that drivers that are built for a particular platform were binary-compatible with operating systems on the same platform, but it also mandated an API, so that drivers would be source compatible to any operating system that implemented the specification, on any platform. That by itself would seem to me to be positive motivation to hardware manufacturers to release the source code to drivers so that they can support operating systems that are on platforms that do not exist yet, or have not been considered (or have been considered to be nonviable or unsupported platforms).

So, I have to wonder why a common device driver specification was never implemented in various operating systems. It would seem to be a common sense thing, especially given that there are so many operating systems. It would make the coexistence of operating systems a lot easier, and it would promote choice. It might encourage bits of proprietary code on free software operating systems, but it would also enable people to drop the excuse that “free operating system x does not support device y”, and would as a result potentially increase the number of free software programs and operating systems in use, even if there is a minor cost in terms of certain drivers. And those drivers could always be replaced—a common driver specification would make it easier to understand the structure of drivers generally, and make it easier for lawful, clean-room reverse engineering to be done on those drivers.

Imagine, for example, if drivers for graphics cards, TV tuner cards, video and audio encoding/decoding cards, modems, storage chipsets, motherboard chipsets, USB chipsets, IEEE-1394 chipsets, graphics tablet devices, touch screens, debugging interfaces, network devices, and so forth were all written to a common specification, it would reduce the amount of code which needed testing. It would increase user choice in both hardware and operating systems—something which I still hold is quite likely the most valuable freedom we have. It would increase reliability, since the users of Windows, OS X, Linux, the various BSD systems, and other, not-so-mainstream operating systems would be able to run the same driver code and collectively supply debugging information and perform testing in a multitude of environments. It would increase security, because then common code that is well-known could be used on all platforms and not just the one it was written for. It would do for device drivers what POSIX has done for user-mode application software. I do not believe that I could be convinced that this would be anything other than a good thing.

Also, it could bring back old operating systems.  Imagine what life could be like, for example, if OS/2 had a “UDI driver” written for it, and it could then take advantage of newer drivers never intended for it. Or any other very old operating system which is no longer supported and could still be useful, for any of a number of reasons…

“You can have freedom without choice.”

That someone could even come up with this one is just amazing to me. Note that this is not an exact quote, but it is the summary of Friday’s topic. For example, this summary comes from the idea that Canonical is bad for considering making mainstream non-free software available for Ubuntu based on user preferences. It does not matter who came up with it, of course, but the important thing is that it be called what it is: patently absurd. The ability to choose is a major part of what freedom—or liberty—is. If you cannot make a choice on a matter, then by definition you do not have freedom in the context of that matter. It is quite simple and self-explanatory. Canonical is seeking to increase freedom here, not take it away. Some people actually want to use non-free software; others may not want to use it, but aren’t aware of alternatives. The latter group of people should have our focus with regard to education (but then we should let them make the choice for themselves!).

Note that I am not one of these people: I would rather use free software because of the liberty it gives me that I have come to expect over the years. But I am not going to tell someone else that they are harming me because they would rather use non-free software that is familiar to them. All I can do is show them that there are free alternatives that exist. I cannot—and I will not—make them use it or make them feel bad for not using it. I may not like proprietary software for a variety of reasons, but I will defend people’s right to use it just as I will defend even a stupid person’s right to spew nonsense by way of speech or written word. In other words, “I disapprove of what you say, but I will defend to the death your right to say it,” or perhaps more appropriately, “I [may] disapprove of what [software you run], but I will defend to the death your right to [run] it.” Even I use a package or two that is proprietary in nature (though it is looking like I will not have to do so for much longer, given the efforts to replace these packages with equivalent free software).

It is worth it to note that by adding non-free software to Ubuntu, the free software that is already there does not change. The mere existence of non-free software within its repositories does not make Ubuntu somehow bad or evil. It would add choices that do not currently exist, and that one such as myself or yourself can certainly opt out of—I most likely would, for the most part, as I do not need to depend on non-free application software, and I only use non-free drivers if I have hardware where anything else is nonviable (and only until there are functional free software drivers). Did you know that Ubuntu has an option in the installer to only install free software? Can you say that for your favorite desktop operating system distribution, whatever that might be?

The response to this idea, then, is that without choice, there is very little—if any, really—freedom. The thing that gives us freedom with free software is that we are able to to download the source code, to review/audit it, to change it to fit our needs or fix a problem, and to share those changes. If we cannot do those things, then it is not free software; see the essential freedoms. But non-free software inside a distribution is not something that should not cause you great consternation even if you are among the most dedicated of freedom advocates, for if you are a true advocate of freedom then by definition you must respect a computer user’s freedom of choice. Remember that we choose to run free software because of the benefits it brings to us; we choose to improve upon free software for much the same reason. Eventually, I think that free software will once again become the norm for computer software, on merit alone, for no other reason than the development, release, and usage of free software is a highly practical solution for many things ranging from library code to application software to complete operating systems. It is worth noting that free content—which is similar in concept to free software, which itself is merely a specific application of freedom itself—also appears to making major headway towards becoming mainstream; it is doing so more quickly than free software is, but there is every reason to believe that free software will follow, for it is already.

An Example

Imagine that you are in a store, because you need some milk for dinner some night. You always get 1 gallon of 2%. But, the store has stopped carrying it, because more people buy whole milk and they were throwing away the 2% milk—demand was low, supply got to be too high, so they just stopped carrying it altogether. You leave the store and head to the next in the same town and you find the same thing there. You have a choice of stores to go to, and you have made the choice to go buy yourself some milk. But there is only one type of milk. You no longer have the choice to buy 2% where you are, and so effectively, your freedom to buy it has been taken away. (Of course, you can make 2% milk from whole milk (and make whole from 2% even, or even butter), but I suspect just as many people want to do that as want to write their own free software that they demand simply must exist, but doesn’t yet).

Now, the point here is that there is more than one freedom in play: the freedom of the store to stock (or not stock) various products, which affects your freedom as a consumer to buy the product you want. In the case of software, and choice, if the software you are running gives you all the choices you want, then it fits your needs. If it does not, then you are not going to be able to use it the way you want. Now you have two choices: you can do the work that it would take to make your desired choice possible, or you can use another system (free or proprietary) that will give you the choice that you want. Many people will choose the latter, especially if they are non-programmers. Though I’ve seen programmers also choose to use proprietary systems for something that they could themselves implement. That is their choice, of course. After all, if you really wanted 2% milk, you would have the same choice: make it yourself, or drive to the next town over which might have it available for you (assuming that there is some in stock and that the stores neighboring towns have not also decided to stop stocking 2% milk).

Ubuntu One: The Reason Behind This

This discussion came up because someone on identi.ca made the claim that Canonical is forcing proprietary software into Ubuntu by way of the Ubuntu One client software. I cannot even begin to state just how woefully incorrect this point of view is. First off: the only thing added to Ubuntu is the ability to connect to Ubuntu One, and the software that was added to Ubuntu do to that is licensed under Version 3 of the GNU General Public License. The claim made in response to that was that Ubuntu One is only partly free software, because the server is somewhere else and has not been released. As we shall soon see, that claim is nonsensical—it depends on an extremely naïve view of how software actually works in order to make sense, really.

So, first things first: Ubuntu One, which was added to Ubuntu 9.04, is not proprietary software. The proof rests in the fact that it GNU GPL v3.0, and we know a priori that software licensed under the GPL is free software, so we do not need to go further on that point.

Now, because the software in question added to Ubuntu is free software, we can read it. The essential freedoms granted to us by truly free software ensure this, and the GPLv3 is indeed a truly free software license because it grants those freedoms. Because we are able to study the software and see how it communicates with the server. Once we know how to communicate with the server, we can write that up and design a server that communicates exactly the same way. From there, it is just a matter of patching the sync dæmon that is in Ubuntu to talk to an arbitrary, Ubuntu One compatible server. To determine how to do that, one need only read the Python source code contained in the python-ubuntuone-storageprotocol and python-ubuntuone-client packages. If you do not know Python well, you might expect to spend several days doing that, but if it bothers you so tremendously that you are going to practically start a flame war over it, you may find it worth it to do so.

Of course, the other side to that is this: if you really want Ubuntu One to talk to an arbitrary server that runs free software, and you want that free software to be written, you can fund the effort to write the free software. Approach a proficient developer somewhere out there on the Internet and ask them how much they’d charge to write a server for Ubuntu One. You might not be able to afford the fund the project entirely, but if you get a number from someone, you can start a coordinated effort to raise the funds. If you are lucky enough to be able to fund the whole project, then do so: it is but one way that you can help provide something back to the community. This does not apply to just an implementation of the Ubuntu One protocol, it could apply to anything that you see that is missing and needs to be created. Or you could spend time learning what you need to learn to pick up the project yourself, if you care for the project that deeply. The most important attribute that a person can have in order to get started with development is motivation—James Westby reminded me of this a couple of years ago, something which I had forgotten.

Perceptions: Another (Possible) Reason

It was suggested to me that another possible reason that people would object to having non-free software inside an operating system distribution such as Ubuntu is that they are afraid that the proprietary options have higher quality, or offer superior features, or provide functionality that is not offered by any existing free software. Thus, they have this perception that by adding such non-free software into a distribution like Ubuntu, people will automatically use and prefer it over free software. This simply is not the case. Sure, some people will use iTunes if it is available on Ubuntu. Maybe many people would. I might even do so, if it were legally available for me to use that way and if it supports the purchase of DRM-free music. However, if there were a free software client for the iTunes store, I’d much prefer to use that. To my knowledge, however, there is no such thing that exists.

If there is not a free software alternative for a non-free component inside a distribution of software, if you are offended by that, then by all means, create a free software alternative for it! As mentioned above, you can start on such a project’s development, or you can look for people that would be interested in volunteering for it and coordinating them, or you can put up funds to pay developers to implement it. If you have money, this can be the easy part: find someone who is willing to accept payment for the service of implementing the free software alternative for whatever it is that someone else has funded, wrote, and released as proprietary software. It is not like free software is developed without cost (and if you think that it is, then you seriously do not understand what free software is or anything about the world of free software and have no standing to be getting mad when a company spends money writing software and does not release it as free software. You can try to write companies that write such software and ask them if they will give you any form of written specifications for the software, or an interface definition, or something along those lines. The worst thing that could happen is that you will be told “no”. And do so nicely, or they’ll be more inclined to tell you “bugger off” instead of simply “no”.

“Allowing users to choose proprietary software is anti-freedom.”

Nothing could be farther from the truth; it is the same, in fact, as the above statement that one can have freedom without choice. For example, if Ubuntu adopts iTunes and makes it so that you can “sudo aptitude install itunes” in the future, that is not a bad thing! How can it be—It contributes to the ability to choose, and thereby contributes to the freedom of the end-user. If you are a die-hard free software supporter and do not want to run non-free software on your system, then there is a very simple solution for you: simply don’t install it.  That is a valid solution to the problem. There are tools already available that can be run as a cron job and report on any non-free software that you might have accidentally (or even intentionally) installed. If you are worried about additional non-free software getting into Ubuntu, then help enhance those tools. Or write a GUI front-end for something like the virtual RMS program and work to get that included into Ubuntu as well, perhaps something that can run every time you login to the computer, or that runs as a persistent process that watches the package database on your distribution of choice for updates and then checks to see if newly installed software is non-free and alerts the user. Of course, it’d be most effective as an opt-in system, and not an opt-out one where it would just be annoying.

There is no way, then, that freedom is actually reduced in this way when another choice becomes available. If iTunes were to be included in the repositories (and I suspect it would be, like the restricted, universe and multiverse repositories, a separate opt-in repository; perhaps simply “proprietary” would be fitting), this does not reduce your ability to choose to run a free software media player and manager like Banshee, or Rhythmbox, or even AmaroK if you are so inclined to run that KDE stuff.

Once upon a time, FUD (fear, uncertainty, and doubt) was the tool of Microsoft. We (the free software world) completely hated it when Microsoft would put out FUD, because we would then have to fight that FUD by way of explanation and demonstration. Well, some time ago, a subgroup of the free software world decided to start using FUD themselves—it was done with Mono, and it is being done now with just a survey asking people what sort of software they would like to see in Ubuntu. Now, those of us who are left who are advocates of liberty—both personal and societal—are stuck potentially fighting two battles. One with Microsoft’s FUD—such as the constant notion that you have to pay for software—and one with the “free software evangelists” FUD, who have even gone so far as to say that people should not use certain types of free software (the one who calls himself “The Open Sourcer” even still today tells people to remove certain truly free software from their systems). The truth is somewhere in the middle, between these two ends of the spectrum.

Conclusion

Back to the point at hand: to say that giving a person a choice is a constraint on that person’s freedom, that is doublespeak.; it is saying that “slavery is freedom,” albeit to a lesser degree than that very melodramatic extreme—it simply does not make sense. The concept just does not make sense unless the words that are used to express the concept are dramatically redefined to mean things vastly different from what standard English dictionaries define them to be. The only reason that one has to try to convince someone that additional choice is a constraint on freedom is to try to convince people of things that are not true; to install fear, uncertainty, and doubt into people. This is the sort of behavior that—no matter what community it originates from—is completely immoral, unethical, and absolutely unacceptable. It’s dishonest, and for those of you who know me personally, you know what I think of dishonesty.

Through something of a comedy of errors, the server suffered some strange software issues that prevented it from working this weekend.  There was a bug in a recent update to the server software (running testing software) and that caused longer downtime than it should have due to various interactions between things on the server.  The good news is that this is mostly fixed.

Additionally, this downtime has taught me that there is yet still more to do in terms of getting the server able to stand back up on its own.  I’ve simplified the server’s setup a bit, and I have to write some scripts and other little glue here and there to tie down some of the things I’m doing so that the server can do things like go down and come back up without issues, all by itself.  Getting that done would be generally a good thing.  First things first, I have to figure out a decently reliable way to shutdown the system without having to do something like kill the containers and not give them the chance to cleanly shut down.  Ideally, there would be some sort of command that could be run on the system that would enable the containers to be shutdown.  This is slightly challenging, because you cannot just chroot into the directory tree that the VMs are running in and kill processes, because things like /proc inside the container aren’t visible to tools running on the host.  Oops.

So, I have some work left yet in terms of getting the server going the way it needs to be again.  After that, I’ll be working on it over the next couple of weekends to try to increase its robustness, and so planned weekend downtime for *.trausch.us and www.mischiefinoverdrive.us can be expected. 

Question 1: What is an operating system?

Here is a (very partial, heavy on the Unix-like systems) list of operating systems:  GNU, FreeBSD, OpenBSD, NetBSD, Microsoft Windows, Mac OS, XENIX, AIX, OS/2, MS-DOS, PC-DOS, FreeDOS.  All of these are full operating systems, though some are more compartmentalized than others.  I hear some saying already, "But GNU is not an operating system!"  Incorrect.  When using the Hurd and a GNU userland operating system stack, that is the GNU system.  I hear others saying, "But you left Linux out!"  Correct, I did: Linux is not an operating system by itself.  Many people are under the impression that the kernel is the operating system, and this is simply incorrect.  A kernel is the core of an operating system, but it is not the operating system itself.

If the kernel is not the operating system itself, what is?  An operating system consists of a kernel and software that is used to make that kernel useful—software that provides the user (or the programmer) with an interface to the kernel.  An operating system in the modern sense comprises a kernel and software that surrounds the kernel.  In the case of POSIX, UNIX and UNIX-like operating systems, this includes the C library, a set of utilities (kill, ps, ls, cp, mv, rm, sh, and so forth), and more.  To compare to Windows, the Windows operating system is not just NTOSKRNL.EXE.  That’s the kernel, but there are a suite of executables and libraries that surround it and provide interfaces to it which make the kernel useful.  While it is technically possible to do, law prevents the creation of a "GNU/Windows" or "GNU/NTOS".  It would not be as straightforward as it is to run GNU on a Unix-like system, but it would be possible (Cygwin is as close as it can lawfully get).

So, when we talk about the FreeBSD operating system, we are referring both to the FreeBSD kernel and the utilities and software that make it useful.  "FreeBSD" includes utilities and libraries and all of that, just like the NetBSD or Windows operating systems do.  But, Linux does not.  When we say "Linux", we are necessarily talking about only a kernel, the project that is the brainchild of a man named Linus Torvalds and the army of programmers around him.

Question 2: What is an OS kernel?

An operating system kernel is the core of the operating system.  However, without the rest of the components that build an operating system, a kernel is useless.  To make matters more interesting, a single kernel can be used as a component in several different operating systems.  This is actually the case for many systems, though it is most prominent in the case of systems that are built around the Linux kernel because there are so many of them.

The lifecycle of a kernel is generally something like this:  Set up and initialize itself, load and initialize basic hardware drivers, kick off one (or more) processes that are required to make the system useful, and then loop, handling things like hardware interrupts and system calls from software.  In the case of a Unix-like system, the kernel attempts to run a program called "init" (usually on modern systems, this is found in /sbin/init), which initializes userland and starts running services that make the system useful.  So a Unix-like operating system provides at least a kernel and an init process, as well as programs for logging in from TTYs (or virtual terminals), manipulating files, processes, and the kernel’s state, and so forth.  If a Unix-like system cannot find an init program, it will halt (panic) or return to firmware where a user could tell the kernel to use an alternate program for init (for example, you can pass "init=/bin/bash" to the Linux kernel, and it will start a single program: bash).

Question 3: Where is the line between an OS and the rest?

To answer this question, we must look at several operating systems, and we must be able to know what the difference is between core software and application software.  The line can be blurred, of course, especially when the core software depends on certain application software.  So in order to more accurately tell the difference, we need to know exactly what the core system uses.  A classic example in the UNIX world: some software is used as system software, and is also available for use as application software.  A system cannot be called a Unix without some of it, and a system certainly isn’t Unix-like without it (and remember: Unix is an operating system definition).

For non-technical people, an operating system is a distribution.  That is, Microsoft Windows is an operating system as it is installed; Ubuntu, Debian, and Red Hat Enterprise Linux are also operating systems.  For those who are more technical, though, the line isn’t there:  the line is somewhere between what is shipped for installation and what is the core system.

Let’s compare FreeBSD with Linux.  A typical FreeBSD system contains the FreeBSD kernel, FreeBSD’s C library, FreeBSD’s kernel management utilities, an implementation of init, and implementations of the Unix utilities (ls, ps, cp, kill, etc.).

However, a typical Linux system contains the Linux kernel, the Linux kernel utilities, the GNU C Library, GNU coreutils, GNU findutils, GNU bash, filesystem-specific utiltiies, an implementation of init (could be sysvinit, Upstart, or any number of other ones), and others.

Question 4: When is it GNU/Linux?  When is it not?  When is any system GNU/anything?

The simple answer: most systems aren’t GNU/anything, but they could be.  Take as an example the project to use the FreeBSD kernel (but not the FreeBSD userland).  The system cannot simply be called FreeBSD any longer: a significant portion of the FreeBSD operating system is removed when its userland is replaced.  The project’s operating system is called "GNU/kFreeBSD", not "GNU/FreeBSD" because FreeBSD was a complete operating system before its utilities were removed and GNU implementations dropped in.  This would include the GNU C library and the typical things you see in a Linux installation.  This also means that it’s a different operating system entirely!  Any piece of software that expects to be built on FreeBSD and assumes a complete FreeBSD system might not even build when using the FreeBSD kernel and the GNU userland.

In the case of Linux:  When you’re using the GNU utilities and core system, it’s GNU/Linux.  When you’re not, then it’s not.  For example, Android is a distinct operating system from Ubuntu.  They both use the Linux kernel, but that’s where the similarities stop.  Android is one example of a distinctly different operating system that shares a kernel with another operating system.  Aside from the fact that Ubuntu is most commonly run on x86 or x86-64 systems and Android is most commonly run on ARM systems, if you sat the two side-by-side on the same platform you’d still have different operating systems that do things differently.  This is because Android doesn’t even use the same C library as Ubuntu; many pieces of software that are stated to work on a Unix-like system such as GNU/Linux or FreeBSD assume a richer environment (closer to that specified by operating system standards).  They would have to be ported to Android to run on Android successfully; that’s a major red flag that says "Oh, that’s a different operating system."

Question 5: Does that mean I can create my own operating system?

Yes.  You can.  And you can even re-use the kernel from any other operating system that is licensed in such as way as to permit you to do so.

Let’s think about this in terms of OS X and what Microsoft could potentially do for Windows, as an example.  Apple used to have its own operating system called "System", which was then called "MacOS", and today we call it "Mac OS Classic".  This system was a cooperatively multitasking operating system that ran software specially built for it.  It was very popular, and it was also not as robust as modern operating systems.  It was pretty interesting, though: it did not have a command-line interface.  Nearly everything that was in the System Folder was the operating system itself, aside from installed fonts, extensions, and system enablers (though often extensions and enablers would patch the operating system in memory, or hook into it in some way so as to attach itself to it).  Apple abandoned that system years ago for OS X.  They took a BSD kernel and wrote a stack around it, reusing some components but using a very different architecture to put things together.  The kernel for OS X is called Darwin, and it’s a very important—in fact, the central—piece of the OS X operating system stack.  But it is not OS X; it is but a component.

How could Microsoft do something like this?  Let’s say that Microsoft went and did something similar, taking the BSD kernel and porting Windows’ software stack to run around it.  They also call it Windows.  Let’s say that they modified the kernel to load Microsoft’s old drivers and that they had incorporated a PE loader and ported their software to this new operating system (but retained API and binary compatibility with their legacy operating system that used a very different composition of kernel and userland, by necessity).  Would this mean that they were running FreeBSD?  No.  It would mean that their operating system were based on (at the very least) the FreeBSD kernel.  Though they could keep the FreeBSD stack and package it as a set of extensions to their new operating system, and claim the ability to run FreeBSD (and in a limited fashion, some Linux, if they retain the Linux compatibility mode of the FreeBSD kernel and system) software.  Now that would be an interesting market position for Microsoft to be in, wouldn’t it?  Microsoft could even make their new system compliant in some way with POSIX and the SUS and get branded as a UNIX® system.  But the new system would be more like Windows than it would be FreeBSD if they didn’t keep FreeBSD as-it-were, since they’d likely replace the core stack to fit their desires and needs, like Apple did.

Conclusion

I hope that this has clarified things a bit.  However, if it has not, a good place to start reading is the Wikipedia articles for "operating system" and "Kernel (computer science)".  They are a long articles, but they make for a great starting point for research and clarification of just what an operating system is.  Both articles look at different operating systems (and their kernels).  As to whether or not you can determine what an operating system is, you will want to become intimately familiar with many different operating systems (and types of operating systems; there are families of them as people create more and more systems both original and reimplementations of others) before you start trying to figure it out.  Look at native code and managed code operating systems—also realize that the distinction of what is the operating system can sometimes be relatively hard to determine accurately.

If you want to see what a bare operating system looks like, though, check out one of these operating systems’ source code:  FreeBSD, NetBSD, OpenBSD.  Each of them in their version control systems has the complete operating system, which is much more than just a kernel.  You can then, should you want, figure out exactly what packages on a typical GNU/Linux system take the place of the remainder of the operating system.  It’d be an interesting exercise in learning—if learning is truly what you’re after, and not just argument.

Many projects that are all about being open and transparent provide mechanisms for their users to get in touch with the developers.  Launchpad has become an excellent place to do these things; Launchpad’s Answers and Bugs components are truly excellent ways to get in touch with people that manage software there.  The system is so good that I’d advocate that every project try to move to it.  Launchpad does have its issues, though they are often quickly fixed (and can be fixed even quicker now that it is free software).  But between Bazaar (a truly amazing and flexible distributed version control system) for code hosting and the other components of Launchpad such as the Answers, Bugs, and Translations components, Launchpad provides virtually everything that a project needs to communicate effectively with its users aside from the project’s Web site.  And users seem to naturally be able to reach out using it.

In the nearly one year that I’ve been working with AllTray, I’ve talked—well, written to—several people who have asked questions and acted in their own ways to work to improve the software.  Development on AllTray has been made much easier just because people are willing to speak up.  Everything from the random “thank you” to “hey, are you going to do this?” or “I’d like to see feature x in the new version” or whatever.  It’s truly great.  I can’t say that every project has users quite like AllTray’s, because AllTray is something of a niche application, but AllTray’s users are a great model for other projects.  There has been virtually no negativity from users and everything has been constructive.

To anyone who manages a project that doesn’t use Launchpad, I’d strongly encourage them to do so.  Its users are great, and as far as management of the overall project goes, it’s very easy for project people to communicate back and manage communications in a single spot.  It’s also great that users can ask a question and that the project can do things like go, “oh, hey, this is really a bug,” or vice versa with bugs→questions.  And if you don’t use Launchpad because you don’t want to use Bazaar (I hear that users of git are pretty dedicated to it; I can understand that, but it’s just not for me—it’s too complex, and I like that Bazaar just stays out of my way), consider using LP for everything but code hosting.  Or consider using Launchpad’s code hosting to mirror your project from git, it does this quite nicely.  Or, find some way to do what Launchpad does wherever you do project management.  It works really well, and it’s a big help.

And to AllTray’s users. A big Thank You.  You guys and gals are great.  Keep communicating!