Today’s post is going to build on the previous two posts, and we are going to discuss the protection of your network. I’d like to start by saying that network protection takes place at multiple levels. In order to secure a network, you must work at all levels. It is absolutely impossible to secure a network by only looking at the network’s core infrastructure or by expecting that one can centrally secure everything. This is absolutely not the case. Security is relevant at all levels. In this post we will talk about security at many levels, including:

1. An individual device, such as a workstation, server, or networked appliance.
2. At the level of a single (physical) network segment, which includes all of the workstations, servers, and networked appliances on that segment. Recall that I discussed what physical and logical network segments are when I discussed bridges and switches in my first post in this series.
3. At the level of a whole subnetwork. For simple networks, only one subnetwork will be present, and so the security of the subnetwork will be the security of the network as a whole. For larger or more complex networks, this is security at the level of each individual subnetwork.
4. At the level of the whole network. This is effectively the same thing as security at the subnetwork level, although the business rules might be different (and thus, any firewalling or other security measures at this layer would also follow different rules). For example, subnets might be allowed to share privileged information or services amongst themselves, but such things might not be allowed to actually leave the network as a whole. Keep in mind that your network is just a subnetwork attached to whatever your upstream network is (which might or might not be the Internet itself, depending on your network structure).

We will also discuss things that are found at various layers of the network, and different types of systems: Client/server applications, Web applications (which are just a specific type of client/server application, when you think about it), authentication and authorization systems, file shares, printers, and so forth.

One At a Time: The Individual Device

Managing a network—especially one that has many network nodes—can be a very time-intensive task. There is much to do. Many of the tasks required to manage individual workstations can be automated (which becomes more and more useful as you have to manage larger numbers of network nodes). We have really only three types of network nodes on our networks:

1. Open systems, which we are able to work with in excruciating detail. These are devices that are running operating systems and software that we have the source code available for, including distributions of GNU/Linux, non-GNU Linux systems (such as those built around Busybox), FreeBSD, NetBSD, OpenBSD, and so forth. These are the systems that we truly have the power to manage literally every aspect of.
2. Semi-open systems, which we can do a great deal with: we can install software and maintain software on such systems, and we can follow the vendor’s requirements and security update procedures. Microsoft’s Windows operating system family and Apple’s OS X line of operating systems are examples of such systems. Also, operating systems that are built on BSD-licensed free software operating systems, but themselves are closed (proprietary) forks without source code available fall into this category. (Of note: Microsoft does make source code for Microsoft Windows available to people who have enough money to pay for the privilege. However, as I understand the terms of their source licensing program, one is not allowed to make modifications to the source code, and therefore is not truly able to manage the system with the same level of power or flexibility as any of the systems in the “open system” category.)
3. Black boxes, which are embedded network devices that we do not have the ability to work with at all. A limited configuration interface might be available. Such devices do not typically allow for the installation of new software, such as updated network stacks, unless the vendor decides to add support for such things. These are mostly network appliances, such as SIP phones and networked printers. There may be ways to work around some of the limitations in these devices, but this depends on the interfaces that the devices make available.

Preferences and idealism aside, realistically speaking almost all networks have systems that are types 1 and 3, and many networks have type 2 systems as well. Most small office networks consists of mostly type 2 and type 3 systems, which type 1 systems usually running things “behind the scenes”, if they have type 1 systems at all. We will focus on types 1 and 2 in this section; most type 3 devices do not have security parameters to speak of and must be protected by the network itself. There are, of course, exceptions; we will not concern ourselves with these. We will also make the assumption that these type 1 and type 2 devices are computer systems of one sort or another: desktops, laptops, servers, netbooks, and some types of smartphones (such as those that run Android, which is a non-GNU distribution built around the Linux kernel).

In order to secure a network, the first step is to ensure that all of the devices that are on the network are themselves secure. This implies that there must be some means by which devices can be rejected from the network if they are not, but we will cover that when we take a look at security at the subnetwork level. Individual device security includes the following items (but is by no means limited to only these items):

1. The security of the operating system kernel itself. The operating system kernel provides critical system services; it is usually tasked with the management of hardware resources. It provides a “gateway” to all of the hardware devices in the computer itself, including audio, video, storage devices, RAM, I/O ports, network interfaces, communications buses, hardware cryptographic engines, and so forth. Most operating system kernels also provide the primitives required to handle networking, beyond talking to the network interface. The Linux, FreeBSD, OpenBSD, and NetBSD kernels all handle IPv4 and IPv6 as part of the operating system kernel itself, as well as things like virtual network adapters (which make tunneling possible) and IPsec. The operating system kernel does an awful lot for us! If the operating system kernel has a security vulnerability (particularly one that permits privilege elevation without proper credentials or that permits a user to crash the kernel, resulting in a denial of service) then it becomes possible for anyone to bring the whole system down, taking all of its software along with it.
2. The security of the core system libraries and software; that is, the userland software that provides the other half of the operating system itself. For most systems that use the Linux kernel, this part of the operating system is provided by the GNU project: the C library, all of the core utilities, the basic shell, and several other things. There are also several userland utilities that are maintained by kernel developers that provide a means by which the kernel can be managed and maintained. All of these core items are part of the operating system (and the interface that it exposes to application software). Application software doesn’t worry about performing tasks that are handled by the operating system, such as managing network interfaces (though application software might be concerned with monitoring such things). What this means is that if there are security problems in a function in the system’s C library, any application software (or indeed, any software at all, including core operating system management utilities) that use that function are potentially vulnerable. This makes it important to keep an eye out for reported vulnerabilities in such things.
3. Finally, there is the application software running on the server itself. If the software running on the server is not programmed with security in mind or has not been throughly audited (and even if it has!) it is entirely possible that there are (or will be) security problems discovered. If you are running software that is created yourself, then you need to be especially diligent and constantly be reviewing your applications for security problems. If you are running software provided by someone else, you should keep up on security vulnerability reports, such as those from CERT’s vulnerabilities database. In fact, if you manage any type of network at all, you should be subscribed to the feed (and regularly keep up on it)  so that you’re informed about new vulnerabilities as information on them becomes generally available.

There are also other mailing lists and feeds around the Internet that are focused on reporting security vulnerability information; I would recommend that you be subscribed to as many of them as you can be, and somehow filter them so that you see only the vulnerabilities that are relevant to the systems that you run.

Hire (or Create!) Help If Needed

If you run a network that has more than one or two dozen nodes, it starts to become difficult to manage it all by yourself; not necessarily because of the number of devices involved, but because of the number of activities that are involved in keeping them secure. Certainly if you manage everything manually it becomes extremely difficult. You have system logs to go through, problems to track down and fix, intermittent issues sometimes from seemingly nowhere. For any size network here are some good things you can do in order to be aware of the “security health” of the systems on your network:

• Monitoring software, such that you can monitor uptime, memory consumption, disk space usage, inode usage (when relevant) and other aspects of all of your individual workstations. Then set the monitoring server to call your attention to any situation that arises that requires human intervention. For example, you need to know if the filesystem on any given system is more than a certain percentage full (how much varies on the system and how quickly the data growth is on that system; some systems might be fine alerting you at 95% full, others you might need to know significantly earlier, say when it is 75% full). You can monitor many different aspects of a computer system, and these can all be early warning signs.
• Aggregate your system logs and event logs in a single place. Have a monitoring system available that can issue alerts when extraordinary events take place. Those extraordinary events might be early indications of a security issue. Learn what’s “normal” for your particular network (every network is different!) such that you can filter out normal system activity. But do not throw away log entries. Archive all of them, somehow or another, since they may be important for forensic analysis later. Aggregation of log files also makes it much more difficult for an attacker to cover their tracks by, for example, overwriting incriminating log entries on a system; perhaps a system’s individual log files are tampered with, but the aggregated copy should be in a safe location and not use the same authentication credentials as other systems on the network, in order to raise the difficulty even further.
• Have regularly scheduled “downtime windows” that can be used for things like applying patches. Humans aren’t perfect, and are sometimes even malicious; thus it is necessary from time to time to apply patches to the software that you run. If all the software you are using comes from a distribution (such as Ubuntu) then this process is made significantly easier, because you can perform all patches and issue a reboot command in a matter of minutes.
• If you have many machines, keep a local mirror of the package archives for the operating system distributions that you have deployed. If you use both Debian and Ubuntu on your network, for example, and you have more than half a dozen or so machines, then it’s a good idea to have a local copy of the package archives for both. You will need a decent amount of space (between 30 and 70 GB for each supported release), but it will significantly improve the time it takes to roll out updates and deploy new systems, as well as save bandwidth (since you will only have to download the updates once, instead of n times, where n is the number of systems you have deployed).

If you get to the point where you are spending all of your time performing maintenance of one sort or another, then it is time to either hire more help—or automate more maintenance. But be careful! There is only so much that can be automatically done without the supervision of a human being to notice that something just isn’t quite right…

Create Update Alerts for “Helpless” Devices

Nearly every network has at least one (and frequently, many) devices that are “helpless”. That is, they are not able to check for updates for themselves. That means that it is up to you in order to check for updates on its behalf. You can do this one of a few different ways. I am a huge fan of saving myself time and automating things where possible, of course. However, that is not always possible; your automation might have to be as simple as “Create an issue in our issue management system every x days to remind me to check for updates for device d“, such that at least then you have the ability to be automatically reminded. You can then check and close the ticket when you find that no update is necessary, or close the ticket when you find an update and create a new ticket that has the goal of actually applying the update.

Audit Updates

When you apply updates, if you have the ability to do so, audit them. See what vulnerabilities they fix. See if the vulnerabilities that are fixed are all of the known vulnerabilities. If you have the ability to do so, review the source code differences so that you can see what the changes are. You never know, you could find a security bug (or other bug) introduced in the patches.

Automate Configuration When Possible

Network security problems at the device level do not just come from bugs in software. They can also come from configuration problems. Perhaps a system is—or a family of systems are—too permissive and do not correctly apply business rules. Perhaps the business rules themselves are flawed such that the translation from business rule to configuration is faithful, but creates a security problem. In any case, do not manage system configuration manually if you can possibly avoid it! If you have a medium to large sized network, odds are that you’re going to have multiple systems that have similar configurations.

Use a centralized system in order to deploy configurations to systems in your network. Also, this way, you only have to back up a single configuration database (this simplifies your needs for recovery later should you become the victim of an attack or some other form of disaster). It also gives you the ability to quickly get back on your feet when a system unexpectedly dies—and the ability to audit change control is extremely useful, as well.

Disable and/or Remove Unnecessary Dæmons/Services

Some operating systems come with a number of dæmons (POSIX systems, including the UNIX family) or services, background processes, or whatever. These things can, for example, perform automatic system maintenance. Or they can open up ports and accept commands. Whatever they do, if you do not use the services in your own network, disable them. Anything that you disable, make a note of, and have your monitoring system always validate that it remains disabled. An alarm should go off if it is ever re-enabled (unless you have to re-enable it for some reason, then you should of course tell your monitoring system to not alarm for it). This reduces the number of vectors available to mount an attack. If you do need such a service, but it does not need to be exposed to the world, use a firewall to keep it in (we will talk more about that when we get to subnet and network level security). If you don’t know what a service is or does, research it—ignorance is absolutely intolerable when it comes to working in security. If you don’t already do research on a regular basis and you are charged with the management of a network, that is something that you should seriously consider changing. Seriously.

Network Segments

Security at the level of a network segment is often synonymous with security at the subnetwork level. Of course, as we have already read, multiple physical network segments can be joined together in two different ways (well, more, actually, but we’ll ignore this for now) such that the multiple physical network segments become a large logical network segment.

At this layer, the most important thing is to know your network well. Understand the physical and logical topology of your network. If you find that there are groups of systems that do different things and they are all sharing one logical network segment, take the time to consider whether or not you should replace a bridge interconnection with a router.

To a certain extent, the security of a network is the performance of the network. Denial of service can result in severe business losses, for example. While relatively uncommon, it only takes a single misbehaving (or penetrated) system in order to bring down a network segment. Consider using routing in place of bridging unless there is some significant advantage to bridging between segments.

Subnetworks and the Overall Network

And here we are.  Security at the layer of a single subnetwork. If you have only one network with zero subnetworks, then you only have one subnetwork: the one that is given to you by your upstream network. Every network (except for the root network) is a subnetwork. Every subnetwork is a network. That’s all there is to it—so really, subnetworks and networks are treated the same way when it comes to evaluating their security.

A network has one or more edges; there can be inner edges and outer edges. Outer edges are gateway points to parent networks; a router lives there and it is usually a default gateway. Inner edges are the same, except they point to a subnetwork that is logically “beneath” the current network. For packets to get to the inner subnetwork from the parent network they must be routed through the current network (in other words, the current network provides transit between the two networks). It is also possible for a network to provide transit between two (or more) outer edges. And just to blow your mind a little further: it doesn’t really matter much to the computer what we call an inner edge or an outer edge. It’s all the same to it; it’s just packets moving around from source to destination via hops along the way. The abstraction of a “tree” structure to the network, or the abstraction of “inner” vs. “outer”, is merely there to make it convenient for us simple-minded humans to talk about and really has nothing to do with how the system actually processes and routes packets.

In order to secure a (sub)network, we must be sure that:

• The only connections allowed from outside the network (e.g., originating from the other side of that network’s edge router or routers and terminating on the inside of that network’s edge router or routers) are those that are allowed by business rules.
• The only connections allowed from inside the network to the outside (e.g., originating from the inside of the network’s edge router or routers and terminating on the other side of that network’s edge router or routers) are those that are allowed by business rules.
• Services that are internal to the network are not exposed to the “outside world” (that is, not reachable from outside of the network’s edge router or routers).

When you have a network that has multiple subnetworks, it can be the case that services on one subnetwork should be accessible to a “sibling” subnetwork in the same administrative domain (and sharing the same parent network as implied by the term “sibling”). In that case, some of the responsibilities for the rules listed above are delegated to the parent network’s edge router(s). Of course, it can get to be far more complex than that, even. Sibling subnetworks could have rules that only permit a pair of siblings to share access to particularly exposed services such that while the sibling networks can access the private services on each other’s networks, connections are disallowed from other sibling networks or even the parent network.

Another way to say it: the design of a network structure can be as simple or as complex as required to support business rules. As long as the logical structure of the network (which, as you will recall, need not precisely match the physical structure of the network) is compatible with the rules that are to be enforced, everything is nice and easy. Where things get difficult is when the design of the network is insufficient to satisfy all business rule requirements. One example of how this can happen is normal business growth; e.g., when new business rules are introduced that are incompatible with the present design of the network. And in that case, the part of the requirement for adopting the new business rule(s) includes a redesign of the network’s logical structure—which might be simplified by using tunnels, for example, to create another subnetwork altogether.

Summary

The tools that we have in our virtual toolbox for network management are many—and they vary widely. In our arsenal, we have:

• The wires themselves (and their physical layout)
• Workstations
• Servers
• Physical network interfaces
• Bridges
• Switches
• Routers
• Firewalls
• Network prefixes
• Logical (virtual) network interfaces
• Tunnels
• Encryption
• Subnetting
• ALGs (Application Layer Gateways) and proxy servers

… and more.

Alone, each one of these tools does nothing significant. Together, they do absolutely amazing things—and can do so in ways that are infinitely flexible (and yet surprisingly simple).

What’s Next?

I plan on writing at least another couple of posts in this series. I believe that I am mostly done with the theory component, and so the next post or two will deal with actual networks. In the next post, I’m going to discuss the layout of the network I spend the most time on: my own. That’s the only thing that I know for sure at the moment. From there, it’s pretty much anyone guess.

If I have left you behind, or if you are confused, please feel free to comment and ask questions on any of the three posts in this series. I will get back as soon as I can after seeing the question.

Until next time.

What is NAT?

NAT, or network address translation, is a mechanism that attempts (and fails, in many cases) to provide transparent access to the Internet for multiple IP-networked devices that can not all have public IPv4 addresses. For example, it is used in homes and many small businesses to provide Internet connectivity to multiple computers on a single connection to the Internet with a single public IP address. NAT was invented in the early 1990s (approximately 1993, if memory serves) in an attempt to delay the exhaustion of the IPv4 address space. It was effective, too, for that purpose—we probably would have ran out of IP addresses ten years ago had NAT not come into existence.

NAT has one major advantage: it enables an entire network to share a single IP address, thus conserving address space. However, NAT comes with a number of disadvantages at different levels. Some of these disadvantages are:

• Increase in network operations overhead. Most types of NAT require the ability to maintain additional tables in memory which support the task of both address and port number translation.
• Having a NAT increases the complexity of a network at the IP layer. This will be discussed in more detail in a few minutes.
• Any NAT device will perform more slowly and consume more resources than a plain router—for most computer systems, this is not a major problem. However, for networks with high-bandwidth Internet connections which are using embedded devices for routing (such as a simple, consumer-grade wireless router) this can be a problem. It can also be a problem for any network that has very old routing equipment that has been retrofitted with the ability to perform NAT, as such devices were not designed with enough processor to handle the additional overhead when the network is at full load.
• In network operating systems which behave as routers, NAT is often implemented in the same area as the firewall. For example, in the Linux kernel, the iptables command is used to set up NAT networking on an IPv4 network, and iptables is the front-end to the Linux kernel’s built-in firewall capabilities. This increases the complexity of the firewall code itself, and can make it more difficult to maintain in general, as well as more difficult to audit for security problems.
• End-to-end communication is broken. The Internet was originally designed with the concept of end-to-end communication; that is, one system on the Internet can converse with another system on the Internet directly. Not only does this simplify network design, but it simplifies the design of network applications, as well (particularly those that require bidirectional communication but do not always maintain a persistent connection and require the ability for either side to reconnect upon some sort of a trigger). Some protocols (such as SIP) have worked around this, but such workarounds can be high-cost as well as brittle.

The removal of NAT from protocol stacks therefore yields a number of benefits, including removing all of the disadvantages above. With the transition to IPv6, NAT devices are no longer necessary. Their additional complexity can go away, and networks all around the world will operate more efficiently and with less latency than they do now. It still takes I/O and processor resources to perform normal routing, but nowhere near as much as it does to perform NAT.

Over the years, multiple types of NAT implementations have been created. I am not going to go into a terribly detailed analysis of them all, but they are:

• Full-cone NAT (or, 1:1 NAT). This type of NAT provides no conservation of the address space; one external IP address maps exactly to one internal IP address. While there are uses for this type of NAT, I can think of no use for it that is not better served by another type of device, such as a load balancer, monitoring and failover, or plain routing.
• Address-restricted cone NAT. This is one of the two most common types of NAT. When a system on the inside (usually using RFC 1918 IP address space) sends an IP packet to the outside, the NAT remembers the IP address, protocol, and port of the internal system and relays it to its destination. The destination system may reply by sending packets from any of its own ports to the NAT on the source port that it sent the original packet from.
• Port-restricted cone NAT. This is the other of the two most common types of NAT. When a system on the inside sends an IP packet to the outside, the NAT remembers the same things as for address-restricted cone NAT, but replies from the destination must come from the same port as the packet was sent out to.
• Symmetric NAT. This type of NAT is similar to port-restricted cone NAT.

It is important to consider that a single NAT implementation may combine behaviors from one or more of these types, and some implementations are extremely configurable in terms of what method or methods are used to perform the functions of NAT. It is also important to realize that NAT breaks many legal network behaviors, depending on the application and the type of NAT in use. Various workarounds have been developed in order to traverse NAT devices for some protocols, and sometimes protocols will change in order to add NAT traversal as a core feature, but the overall effect is that NAT (often significantly) reduces network efficiency.

What is NAT not?

NAT is not a security mechanism.

Let me repeat that: NAT is not a security mechanism.

One more time: NAT IS NOT A SECURITY MECHANISM.

I am uncertain where people have gotten the idea that somehow NAT was designed to increase security. It was not. It was designed to help conserve the increasingly scarce resource of IPv4 address space, nothing more. It is not a security tool, and it does not provide any additional security over a properly maintained IP firewall—using a firewall is essential with or without a NAT in place if you need to do any sort of packet filtering at all.

The idea that NAT is a security mechanism probably came from the notion that one cannot see the addresses on the inside of a NAT. However, there are many mechanisms by which a sufficiently interested attacker would be able to determine things such as an approximate (or even an accurate!) count of how many devices are behind the NAT and what their IP addresses are through the use of various protocols, application tricks, and security exploits. For that matter, it is trivial to do things such as setup IPv6 on a NAT’d network and give all the systems on the NAT’d network a globally reachable IPv6 address, all without the cooperation of the NAT device. Only a firewall can stop such a thing, and only if you know what it is that you are trying to stop. And there are some things that even a firewall cannot protect you from (such as trojan horses and intentionally malicious employees and ex-employees). Security is an insanely complex problem to solve, and NAT is not a tool in a security professional’s toolbox.

How does NAT work?

A NAT device has a fully functional IP stack, and operates at a combination of OSI layers 3 (Network Layer) and 4 (Transport Layer)—mostly layer 4. In comparison, a router is an OSI layer 3 device. (In case you haven’t memorized the OSI model yet, refer back to my previous post which shows it.) Let’s say that you have a computer system that is on an IPv4 network, and that IPv4 network is using NAT. When you came to my blog to pull up this post, your Web browser performed the following tasks:

1. It looked in its DNS cache to see if the hostname mike.trausch.us was there. If it was, it used the IP address from the cache; if not, it asked your computer to find the IP address for mike.trausch.us.
2. Then, it asked the operating system to open a TCP socket connected to the IP address for mike.trausch.us on port 80. Since you are reading this, it is probably safe to say that it succeeded, and it was given a socket to work with.
3. It then asked my Web server for the post, which my Web server kindly gave to you. There are a few back-and-forths that I am omitting here for the sake of clarity.
4. The connection from your computer to my Web server was then closed.

In a normal (that is, non-NAT’d) network, this was all nice and direct, and the edge router on your network made it possible for your packets to get here. Even better, in such an event, the chatter in such an event is directly between your computer and my Web server. However, on a NAT’d network, this is not the case. Instead, this is (some of) what happens:

1. Your browser looked in its DNS cache to see if the hostname mike.trausch.us was there. If it was, it used the IP address from the cache; if not, it asked your computer to find the IP address formike.trausch.us.
2. Then, it asked the operating system to open a TCP socket connected to the IP address for mike.trausch.us on port 80.
3. Your computer’s operating system opened the socket, but not to my Web server. It just thinks it did. When the packet went out that was supposed to start the TCP handshake, it ended up at your NAT.
4. Your NAT sees the packet, and makes a note of what the source IP address (your private IP) and source port was.
5. The NAT notes the (source IP, source port) pair, and notes the destination address (e.g., the IP address for mike.trausch.us in this instance), the protocol (in this case, TCP) and sometimes also the port number (in this case, TCP port 80).
6. It then forwards the packet to my Web server on port 80.
7. My Web server receives the packet, which at this point appears to come from your external IP address, and probably a different port from the source port on your computer.
8. My Web server sends a return packet, directed at your NAT device’s IP address and the port that it sent your packet from.
9. When your NAT receives the packet, it looks in its table of entries to see if it has a mapping for the IP address and port it received a packet on.
10. It then forwards my acknowledgement to your computer.
11. All of the rest of the steps are the same, but with the NAT intercepting, looking up, and rewriting every single packet before it is shipped to its destination (either your computer or my Web server).

It is a lot more complex to do all of this, obviously.

And for larger networks, it won’t scale at all: multiple external addresses will have to be used to represent the whole network, because each NAT device can only have a mapping for one system and one (IP, port) combination at a time. What that means is that for larger NAT’d networks, you might have a different “public” IP address for every connection—or if you have a large network and only one external IP address to NAT with, you might actually wind up sometimes not being able to connect to anything at all, because the mapping table in the NAT is full.

So I Have to Learn to Use a Firewall?

Yes. Well, no. Well, sort of.

Consumer devices for use in home networks that support native IPv6 will most likely be running their own firewall with a reasonably sane default set of rules (and hopefully, the ability to change those rules!). For example, not allowing inbound packets to protected ports (those below 1024) and ports well-known to be open and accessible by default by operating systems in the Microsoft Windows family. Of course, it will also be up to people to not install services on their computer systems that are configured to service the world. That’s not terribly hard, given that we have the loopback network (127/8) that is reserved for use locally (such IP addresses aren’t even allowed to reach the physical network outside of a sole system).  This means that a system can run services that aren’t to be exposed to the world (for testing, or in order to protect them from access without first using an SSH tunnel, or for any manner of other things).

Very small, client-system-only networks will most likely use consumer-grade devices, as well, and need not worry about it for the same reasons that home networks won’t likely need to worry about it.

Power users, network administrators, and everyone else in between can instead just configure a firewall. Whatever device is providing core routing functionality more likely than not has the ability to perform firewalling as well—and if not, it’s easy to obtain an operating system and a computer that can perform the task. After all, Linux, the BSD family, and most other UNIX and UNIX-like operating systems not only can function as routers, but can firewall (and Linux and the BSDs are free). If you are the administrator of a network that has more than ten nodes on it, or the administrator of an any-sized network that has more than zero server systems on it, you should know how to use a firewall both in general and the particular implementation that you have.

What About RFC 4193 (IPv6 Private Address Space)?

RFC 4193 does indeed provide for private address space in IPv6. That does not necessarily mean that it has to have NAT. Private address space can be used to ensure that one or more subnetworks have absolutely zero Internet connectivity (or can use something such as an IPv6-enabled SOCKS server in order to have strictly controlled connectivity). However, more often than not, servers that require such security need not use address space reserved for it by RFC 4193, as it would unnecessarily complicate the network. Instead, one could use a single subnet out of their allocation of subnets, treat that subnet as “dark” or private, and configure the firewall to prohibit all (direct) communication with that subnet. If you have a correctly configured network this is trivial.

Private address space in IPv6 can also be useful to create disconnected islands, or testbed networks. However, in production networks, I would expect to see a business that has a /48, for example, simply devote a single subnetwork to private-use.

There are other means by which one can protect their network without NAT; see RFC 4864 (“Local Network Protection for IPv6″) for more information.

So, no NAT in the future?

That’s right. We are heading to a world without NAT, a world where no NAT is needed, and a world where the overhead of the Internet as a whole will be reduced as a result. That pretty much wraps it up for today’s post. Questions? Comments? You know what to do with ’em!

Before you get too deep in here, note that you’re likely to encounter things that you know already in it. But, it should be useful for anyone attempting to learn networking from the ground up. Most people in IT who are relatively new to it (or have been shoehorned into a position of supporting a network) have only really learned about bits and pieces of it, often as a result of being practical about learning only what is needed to get things done. As a result, I don’t really know how much “below” IP people who read this post know about. Even if you know about all of this stuff, I’d appreciate a full proofreading.

All of the IP addresses that you will find in this post are pulled out of pools reserved for documentation and examples. None of the IP addresses you find in this post will work on the public Internet. It might be possible to use them in isolated network islands, but there is network space reserved for use on islands; see RFC 1918 for IPv4 and RFC 4193 for IPv6 for address space reserved by IANA for such purposes. The MAC addresses in this post technically come from Xerox’s pool of MAC addresses. Don’t use them anywhere. I could not find a MAC address prefix dedicated for documentation/example purposes. If someone happens to know of just that, please let me know.

There are two network models: the four layer IP model, (I am calling it the IP model and not the TCP/IP model because I won’t carry forward the misnomer “TCP/IP”, as TCP is far from the only thing that runs on IP) and the seven layer OSI model. I use the seven layer OSI model in part, because it is the model that I was taught when I learned about networking, but also in part because it is more descriptive and has a finer level of granularity—it is easier to talk about more layers that are smaller in scope than fewer layers that are larger in scope—at least, IMHO.

The OSI Network Model

The OSI model (Open Systems Interconnection model)—sometimes called the ISO network model since it was defined as an ISO standard—is the basis for the discussion that I will use in this post (indeed, it is the basis that I use for all of my discussions on networking). The OSI model has been around for a very long time, dating back to at least 1980 and probably even further. I wouldn’t know without a lot more digging, since I wasn’t around when it was created.

The OSI model has seven layers. Conceptually, each layer only communicates with the one layer above and the one layer below it (with exception for layer 1, which has no layer below it, and layer 7, which is the top). The seven layers of the OSI model are:

1. Physical layer. In your average Ethernet network, this is essentially the cable and the transceivers at either end of the cable. For dial-up networking, it consists of the serial lines at both ends, the modems at both ends, and the telephone network in between. This layer of the OSI model is concerned only with the movement of bits across the physical medium and passing data to and from layer 2. This is the hardware-specific layer and it is spoken of in terms of electrical connections, circuits, and signals.
2. Data Link Layer. In your average Ethernet network, this is the frame-based protocol (also called Ethernet!) that is used. For dial-up networking and networking on cell phones, it is (most often) PPP. This layer receives bits from layer 1, interprets those bits as frames, strips the framing off, and sends the frame’s payload data up to layer 3. The inverse operation is performed when data is received from layer 3: framing is added and the bits are sent down to layer 1.
3. Network Layer. No longer do we care about what type of physical network is in use: layer 3 is where IPv4, IPv6, ICMP, ICMPv6, IGMP, IPX, AppleTalk, and so forth all live. The network layer is responsible for packet routing, filtering, and delivery. It receives data from layer 2, strips the network protocol framing off, and sends the data up to layer 4; the inverse is taking data from layer 4, adding network framing, and passing the packet down to layer 2.
4. Transport Layer. This is where TCP, UDP, SCTP, and others live. The transport protocol determines whether the transport is connection-oriented or not, whether it sends streams or datagrams (or can do either or both), whether or not reliability is provided, and so forth. This is the layer used by application software using the Berkeley sockets API when talking directly to a socket (in truth, the BSD sockets API also exposes a bit of the Network Layer, so it’s not a pure Transport Layer wrapper).
5. Session Layer. This layer is not often discussed in relation to IP networks, because it considers the duties of layer 5 to be part of its application layer. However, the presence of layer 5 is useful as an abstraction when talking about sessions and how they can be thought of, and libraries that provide a sessions support mechanism on top of the BSD sockets API effectively implement the OSI Session Layer for applications that talk over IP. NetBIOS and L2TP are two such protocols which could be considered to be in this layer (and L2TP is an example of “a network in a network” in terms of the OSI network model, but we will skip discussion of recursive networking for later: even though we use it all the time in IT, most people don’t realize it and think the concept is more confusing than the application).
6. Presentation Layer. Like the Session Layer, the TCP/IP model considers the duties of layer 6 to be part of its application layer. MIME, SSL, and TLS can all be considered to be OSI Presentation Layer components. The OpenSSL library effectively implements SSL (and TLS) as a layer by wrapping the BSD sockets API.
7. Application Layer. This is the layer concerned with actual applications: DHCP, DNS, FTP, HTTP, POP3, IMAP, IRC, XMPP, Telnet, SSH… the list goes on and on. The application layer represents application protocols that we use every day for IP address provisioning, turning names into IP addresses, transferring files, browsing the Web, email, chat, remote control of computers, and so forth. Many of the things that we consider application protocols in the context of an IP network also implement things elsewhere: for example, SSH is an application protocol in IP networking that performs tasks at OSI layers 5, 6, and 7.

As I mention in layers 5, 6, and 7 in that list: the IP model of networking essentially merges those three layers together into a single layer. It also merges layer 1 and 2 into a single layer. Because of that, it is harder to focus on a single precise task in the IP model. However, it is worth mentioning that even in the IP model, the Internet Layer (equivalent to OSI’s Network Layer) and the Transport Layer (equivalent to OSI’s Transport Layer) are separate things.

As you can imagine, the layer looks like a stack. This is probably where the term “network stack” comes from; software implementations often use the layer model, resulting in a suite of software “stacked” one atop another. In theory any one layer’s implementation can be altered or completely changed, and as long as it preserves the interface it has with the one layer below it and the one layer above it, nobody should really notice. In practice, it works almost as smoothly: applications that use TCP for transport are able to work on IPv6 as well, usually with few changes to the source code in order to select an IP-version agnostic socket type (if, that is, any changes are required at all). This is also why IPv6 support is nearly ubiquitous (at least, in popularly used free software).

An Overview of OSI Layers 1 and 2

Before we can continue on to discussion of the many things that IP does for us, it would be helpful to take a closer look at the OSI Physical and Data Link layers. These are the layers upon which both IPv4 and IPv6 build in order to provide us with the convenient networking that we are so used to today. We will also take a look at some of the network equipment that operates at these layers, so that we can understand how they fit into our networks. For the purposes of this section, let us imagine two computers (we will call them A and B) which use a standard gigabit Ethernet (1000BASE-T) to connect to each other and provide a network. As we are limiting our discussion to OSI layers 1 and 2, in this example we will not care what is handling the jobs of layers 3 through 7.

The Physical Layer provides the medium upon which communication can take place, as well as the electronics which directly use the medium. So, if we look at computer A, we have the following things that represent the Physical Layer:

• A Category 5 cable. The cable is the actual transmission medium: data “flows” over the cable.
• The Ethernet port, (an 8P8C modular connector) which the cable is plugged into. The wires in the cable connect with wires in the port, which run to the Ethernet interface’s transceiver.
• The Ethernet interface’s transceiver, which is responsible for transmitting (and receiving) binary digits (bits) to and from the wires in the cable.

Operating systems are not concerned about the physical layer in a computer network; that is a job for hardware to handle all on its own. (This is not the case if the Ethernet interface offloads its processing to the host system’s CPU, but we will disregard this as an unnecessary complexity for the time being.) In this specific example, the role of the Physical Layer is to receive data from the Data Link Layer, serialize that data into bits, and transmit it over the cable. Likewise, bits that it receives from the cable are deserialized and sent up to the Data Link Layer. At the Physical Layer, the only thing defined is the connection between two (or more) nodes on a network segment as a bus or star network. Some examples of layer 1 network devices are ports, connectors, cables, terminators, and hubs (despite the link being to “Ethernet hub”, other types of networks can have hubs, too; they’re not specific to Ethernet networks). A hub is just a device that has more than two ports for network devices to plug into, and everything that is transmitted on the hub is sent to all of the ports at once; effectively, a hub is a repeater and connects all of the nodes together as if they were all sharing a single cable, but they are not. Bridges and switches, two types of popular networking equipment, are not OSI layer 1 devices as we will soon see.

The Data Link Layer (or, layer 2 in the OSI model) can be thought of as providing two “sublayers”: Logical Link Control (LLC) and Media Access Control (MAC). The term MAC address refers to the address that a particular piece of network equipment uses at the MAC sublayer of the Data Link Layer. LLC provides the ability to multiplex many different network protocols over the same physical medium, while MAC provides an address space as well as the rules by which transmission and reception over a shared medium work. Think about a network segment as a town, and a MAC address as a street address within that town. It’s possible to have multiple street addresses that are identical as long as they are in different towns, but if two places have an identical address in the same town there are problems such as incorrectly delivered or lost mail. Instead of lost mail, a duplicate MAC address on a single network results in frames not being delivered, being delivered twice, or being delivered sometimes to one or the other of the nodes in conflict. The behavior in such a situation can be extremely erratic. It is only a common problem in environments where humans set MAC addresses, such as when using certain types of virtualization software.

The Data Link Layer is the layer that handles addressing, framing, and sharing of the network segment. It is still not possible to do true internetwork routing using just the Physical and the Data Link layers. You can, however, do two new things at layer 2: network bridging, and packet switching. In fact, every network switch is logically just a multiport network bridge where all of the ports are bridged together—it is important to realize that. This brings with it two benefits: more collision domains (and thus ability to increase the usable bandwidth on a network segment given certain types of traffic patterns) and the ability to join multiple physical network segments together into a single logical network segment. Also, despite a switch (or a bridge) being a layer 2 networking device, they are invisible to the networks. A bridge behaves as an intelligent, transparent proxy between two networks, turning two physical Ethernet segments into a single logical Ethernet segment; a switch is nothing more than a bridge with multiple ports. Yes, that does mean that if you have an 8-port switch—and you’re using all of them—what you have is 8 distinct physical Ethernet segments, but it behaves as if it were a single Ethernet segment, so it is a single logical Ethernet segment.

On today’s networks it is very rare to not have a switch somewhere, and while bridges are not exactly commonplace they are used quite a bit. Bridges and switches both make it much easier to manage a large network as if it were a single large network, but without many of the problems that large networks would generate without them. Prior to the widespread availability of inexpensive bridges and switches the two primary networking tools were hubs and routers. Networks were forced to be compartmentalized in order to keep the size of its collision domains manageable. There were also limitations in the length of cable that could be present between two nodes that wanted to communicate with each other.

Switches do not entirely eliminate all of these pains, though. There is a limit to the length of a cable in an Ethernet network, and there is a limit to how many switches may be present between two systems who wish to talk to each other on a logical subnet. If you find that you need to exceed those limitations, your only option is to use subnetworks and add a router to the setup.

Enough about the Physical and Data Link Layers! Back to the Computers!

At the beginning of the last section I mentioned two computers: A and B. We now have enough information to talk about how they communicate with each other. Computer A would like to say “Hello!” to computer B. Computer A needs to know computer B’s address on the network (or needs to know how to broadcast over the network) in order to do so. For now, we’ll just assume that we want to broadcast—it makes things more simple! In terms of the Data Link Layer, the broadcast MAC address is FF:FF:FF:FF:FF:FF, or the “all F” or “all ones” address. Any layer 2 packet sent to this address is received by all systems on the network (though it may not be processed by all of them). Let us assume that computer B is listening for broadcast messages, and whenever it sees “Hello!” sent to the broadcast address, it says “Hi there!” back, also sending to the broadcast MAC address on the network.

Now, let’s say that computer A has the MAC address 00:00:01:00:00:01 and that computer B has the MAC address 00:00:01:00:00:02 (no, that would almost certainly not happen on a real network). Computer A knows that it wants to address computer B with “Hello!”, so it sends “Hello!” to address 00:00:01:00:00:02. The frame is transmitted on the wire, computer B’s network interface sees the frame and because of the address decides to read and process it. It then generates a new frame, addressed back to 00:00:01:00:00:01, which says “Hi there!”, and the exchange is complete.

It is important to keep in mind that there is sufficient power at this layer to perform only simple networking: unicast or broadcast on a single network segment are the only possibilities. Therefore, it is not suitable for internetworking by itself. In order to achieve internetworking, we have to have the ability for computer A to talk to computer C, even when computer C is not on the local link, perhaps on the other side of computer B (in which case, computer B would be a router). However, a router is not a layer 2 concept: it is a layer 3 concept. And so, guess what we’re going to talk about now?

OSI Layer 3: The Network Layer

It is at this layer that it becomes possible to perform internetworking. That is what makes the IPv4 and IPv6 protocols so special: they enable the ability to have a network so large that no one can completely imagine it.  A network is composed of smaller networks that participate in providing transit for it, which consist of even smaller networks, which themselves may have networks (and this recursion can be nested arbitrarily deep; there is no limit on the depth of the tree that is the Internet). [question: would a diagram here showing the structure as a tree data structure be useful here?] This means that you may build networks consisting of different physical network types, and they can still be linked together and participate as a single, harmonious (we hope) network.

In fact, this is precisely how you are able to be on the Internet reading this at all. Your computer has either an IPv4 or an IPv6 address on the public Internet, or it has a private IPv4 address on a network that is essentially transparently proxied to the Internet. Either way, your computer was able to reach my computer, and request that this blog post be transferred to you in order for you to display it. When I think about all of the things that make this wonderfully huge network actually work, it is hard to imagine—even though I have been on the Internet for roughly two-thirds of my life.

Let us now expand our example network from earlier. Imagine computer A has two Ethernet ports. A cable is plugged into the first Ethernet port, and the other end of that cable is plugged into computer B. Yet another cable is plugged into A’s second Ethernet port, and is plugged into computer C. If you visualize the three computer systems, there is A, which is the vertex of two lines that are the cables in its two Ethernet ports, and computers B and C are at the ends of those two lines (said perhaps more simply: if B and C had a cable connecting the two to each other, we would have a triangle). We still have no connection to any other computer network, including the Internet, in this example.

At this point, what we want is for computer B to be able to say “Hello!” to computer C. But we have an obstacle in the way: there are two network cards in computer A, and B is attached to one while C is attached to the other one. What we have is two network segments, and all of the following statements are true:

1. Computer B is on one network segment, connected to A.
2. Computer C is on one network segment, also connected to A.
3. Computer A is on two network segments, one of each is connected to computers B and C.

Since we are string for an example in routing, computer B and computer C will be on different IP networks; computer A will be on both of these networks. This means that:

1. The first Ethernet interface on computer A (which we will call eth0) will have the IP address 203.0.113.1/29.
2. The second Ethernet interface on computer A (which we will call eth1) will have the IP address 203.0.113.9/29.
3. Computer B’s eth0 will have the IP address 203.0.113.2/29 and it will be configured with a default gateway of 203.0.113.1.
4. Computer C’s eth0 will have the IP address 203.0.113.10/29 and it will be configured with a default gateway of 203.0.113.9.

We could say that we have two sibling networks. That is, we have two networks, neither of which are a physical subnetwork of the other one. Since we are disconnected from any other networks, this addressing scheme works just dandy for us. And, with a suitably sane operating system, we have nothing more to do, either: all three of these computer systems ought to be able to talk to each other just fine now, because:

1. When we added the IP address 203.0.113.1/29 to eth0 on computer A, it should have put an entry in its local routing table that 203.0.113.0/29 was reachable via eth0.
2. When we added the IP address 203.0.113.9/29 to eth1 on computer A, it should have put an entry in its local routing table that 203.0.113.8/29 was reachable via eth1.
3. Computer B was configured with an explicit default gateway, and that gateway is set to point at computer A.
4. Computer C was configured with an explicit default gateway, and that gateway is set to point at computer A.

There is one catch: the Linux kernel will not behave as a router if it has not been configured to do so. That is, it won’t forward packets arriving out from eth0 but obviously destined to eth1 to eth1; instead it must be told to do so. I suspect that other operating systems have similar settings that must be flipped in order for the system to become a router. Once that is enabled, the routing table is all that is necessary in order for the kernel to make routing decisions. An added bonus is that if the kernel receives a packet destined for a network that it knows it cannot connect to, it will send an ICMP reply indicating that the destination network is unreachable (e.g., there is no route to the requested node). So, if you’re trying this out at home, you may have to enable some form of IP forwarding for things to work, but they’ll work. Computer A will behave as a router.

In terms of network diagnostic tools, B should be able to ping both A and C; C should be able to ping both B and A; and a traceroute between B and C should show A as the first hop and C as the second hop. We have a functional, routed, network! However, this also means that network broadcasts on computer B’s network will not reach computer C’s network (and vice versa). That is to say, computer C is outside of computer B’s broadcast domain: they can both broadcast such that computer A will see the broadcast on its respective interface, but they cannot broadcast to each other. (There is something called multicast, and if computer A were configured as a multicast router, systems on both networks connected to A could subscribe to multicast groups managed by A and thus could send things to the multicast group and computer/router A would pass them along to the other network. However, that’s pretty advanced and we’re going to ignore that for now.)

Great, but how does all that work anyway?

So we know now that computer B can talk to computer C—even though there is no direct physical connection between computers B and C. We know that this is done because A is set up to route packets between B and C. So, how does an Ethernet packet from B wind up getting to C? The answer is that it does not. Huh?

Right. It does not. Because routing is a layer 3 thing, it is layer 3′s job to figure out what the next hop is for the IP packet. If computer B is attempting to send to computer C, it knows that it cannot do so directly. Computer B knows this because it is only attached to the 203.0.113.0/29 network, and 209.0.113.10 is not in 203.0.113.0/29. So it looks to its default route (that is, the system specified as the default gateway) and figures out what the next hop’s IP address is.

Here, we are going to stop for a minute and figure out how layer 3 gets anything done. When two computers are on the same IPv4 subnet it is simple: if computer A wants to send to computer B, computer A makes an Address Resolution Protocol (ARP) request; this is a broadcast, asking the network “who has the IP address 203.0.113.2?” and expecting a response. It will try this a few times before giving up; if it does give up, then the host is deemed to be unreachable: either no systems on the network have that address, or something is malfunctioning and it is not receiving, not transmitting, or otherwise impaired. (An aside: this is one thing that works differently in IPv6; instead of using ARP, IPv6 uses Neighbor Discovery Protocol (NDP) to find a host on the local network.)

So, when computer B attempts to talk to computer C, here is what happens:

1. Computer B notices that 203.0.113.10 is not in its own network (which is 203.0.113.0/29).
2. Computer B looks in its routing table, and it sees that it does not have a route to a network that contains 203.0.113.10.
3. Computer B therefore looks again in its routing table to see if it has a default route. It does.
4. Computer B looks up the address for the gateway specified in the default route (203.0.113.1).
5. Computer B makes an ARP request to get the MAC address for the node with the IP address 203.0.113.1.
6. Computer A, having the IP address 203.0.113.1, sees this ARP request and says “I have that IP address.”
7. Computer B sees the reply from computer A, and makes a note of its MAC address.
8. Now, computer B takes the IP frame as a sequence of bytes and sends it down to layer 2, and tells layer 2, “Give this packet to this MAC address,” and layer 2 does its thing. The packet makes its way to the wire destined for the MAC address of the interface eth0 in computer A.
9. Computer A receives the layer 2 frame, reads the bytes from it, and passes it up to layer 3.
10. Layer 3 (IPv4) on computer A sees that the destination of the IP packet is not computer A.
11. Computer A now proceeds to perform steps 1 through 7 again, except that computer A is looking for computer C. Computer A may broadcast the ARP request out of both of its interfaces.
12. Computer A now sends the packet to computer C.
13. Computer C, seeing that the packet is destined for itself, sends the packet further up the stack, eventually hitting the application layer, where a listening application may do something with the data.

That’s a lot, and it happens very quickly. Each of the computers will have an ARP cache, which stores the MAC address↔IP address mapping for a limited amount of time. When the entry goes stale, it will drop from the ARP cache, and the next time the computer has to talk to that IP address again it will issue another broadcast for the MAC.

Also remember that during all of this, layer 2 devices are invisible to the layer 3 protocol. Switches are smart enough not to interfere with the operation of things like ARP, by transmitting all broadcast packets out of all active ports (except for the port of origin); the same goes for all other types of MAC broadcast traffic. This is why DHCP works even when there is a switch between the computer requesting an address using DHCP and a system running a DHCP dæmon: the client initially sends the request to the special 255.255.255.255 IP address, and IPv4 stacks know that 255.255.255.255 is a network-wide broadcast address, so it in turn tells layer 2 to send the packet to the special FF:FF:FF:FF:FF:FF broadcast MAC address. Any bridge that sees that MAC address as the destination will send the packet across the bridge; remembering that switches are just a special application of a bridge, it does the same thing.

What next?

I am going to leave this as it is tonight. This concludes the first part of my post. You should feel comfortable with the concept of how the lower layers of networking work, and with how the Internet Protocol adds functionality to those lower layers of networking. If you do not feel comfortable with that yet (and you have read all of the links to Wikipedia and done research from there on out), then I want to know what your question(s) are.

I am not sure when I am going to write the next post in this series, could be tomorrow or in a week. In the next post, I will add more complexity to the network. There will likely be diagrams, because it won’t be simple enough to unambiguously describe in plain text anymore. After the next post you should be able to feel comfortable with diagrams, though, at all of the first three levels of the OSI model (and realize that they can—and very often are!—very different from each other).

Perhaps one key point to take from this post: Any time you have more than one logical network segment that is not connected by a bridge, you must have a router—and almost any computer operating system (certainly anything from the UNIX family) can be a router just fine.

That is perfectly fine. I’m not here to be a politically correct person. This is my space on the Internet—and I’ll say what I like here. You know how I believe in freedom and its natural limitations? No? Well, I’ll say this until the day that I die: one person’s freedom ends where another person’s freedom begins. I’m free to say what I like, and you’re free to read it in any way you like or even not at all. Now, I won’t talk about other people by name on my blog unless I have permission to do so—especially if I’ve nothing positive to say, so don’t say that I’m not being at least a bit nice to you—but if I write a post here and it’s about you and you know that from nothing other than reading that post, maybe that is saying something more than what I have said here. If that has ever been the case in the past, or if it ever becomes the case in the future, then know this: it’s better you read what I have to say about you here, than wish that I were willing to spew my unfettered anger in your direction in person. And shoot, if you’re doing things that are illegal (such as running unlicensed proprietary software) and I haven’t turned your ass into law enforcement, you should be thanking whatever deity you believe in. Honestly.

Anyone who knows me—even if they do not know me well—knows that there is absolutely nothing that will make me angry like willful ignorance.  “Ignorance is bliss,” as the saying goes, but ignorance when combined with the lack of desire to fix it in what you claim to be a domain of specialized knowledge which you possess is just plain inexcusable. If you manage a network, you should know the basics of how it all works. There are people that I know that manage Windows networks and know nothing—not even the high level overview—of how Windows networking actually works beyond the painted pixels on the screen. Guess what? That means that you really do not know what you’re doing.

And lest anyone get offended or butt-hurt over being called ignorant, don’t. Let’s remember what ignorance actually means, and remember that it’s not an insult or a sleight against anyone—everyone is ignorant about many things, even in their own fields of work and expertise. However, willful ignorance in one’s own field—that is, ignorance that you’re not willing to fix all on your own like a big boy or girl—is absolutely something that you should be offended at! It is simply not possible for a single human being to know everything, even in his or her own specialty.  We have reference works, documentation, and vast seas of information in every field that I can think of, more than can fit in a human brain. But what matters is that you know what you know, and know what you don’t, and know how to find it out quickly and efficiently. And that means having a sort of self-initiative. And for that matter, I’ll even point people in the right direction, if they’re willing to do the legwork themselves to decipher the information once I’ve pointed them at it. I certainly don’t spoon-feed though, and if you expect that (in your own field, no less!) then I will stand by my assertion that you should not work in that field at all. And I will stand by that assertion whole-heartedly, no matter how much that gives a person pain.

If you’re ignorant about something that you never do nor have a desire to do—say, you’re an auto mechanic and you don’t care to know how to sew or crochet—then that’s fine; that’s your choice! But if you work in a field, and you learn that there is something that you don’t know, then learn it. Or at least learn where you can learn it when you need to, and get a friggin’ overview in your head. Read any single RFC and you’ll realize that there is no way that any of us can memory every single detail of every single specification for every single type of system that we manage. It’s just not possible without spending so much time studying that as to make it impossible to get anything useful actually accomplished. But if you manage a mail server and you don’t know the first thing about SMTP or POP3 or IMAP or whatever-else protocols your mail clients and servers are using, yes, that’s a problem. You certainly do not need to be able to have a conversation with your SMTP server, but you should know how to look up just how to do to that should you ever have to do any really low-level troubleshooting or log capturing. You shouldn’t need to know how to speak any application layer network protocol directly for that matter (though a lot of the text based ones are simple enough that you can learn them as needed over time). But you absolutely should know how to find the information that tells you how to speak those protocols if ever you have a need. And you should know enough to be able to make intelligent decisions on things like physical network infrastructure, management of your client and server operating systems, and so forth.

As an example: I am nowhere close to an expert on Windows—and I know this. (I will say that I know an awful lot about the way Windows bootstraps itself, as I have had to fix systems with multiple infections by hand because there were no automatic tools available to fix the system… but that does not make me an expert on the whole system, and probably not even the bootstrapping process of the system.) But I will research any issues that I encounter while supporting Windows users and find out—empirically, if I must—how to fix the problem. It’s what I do. And there are many, many places where I can find that sort of information, including booting up a copy of Windows itself and trying to figure it out that way. It does probably take me a lot longer than it would take someone who knows the system in and out, and I’ll grant that. I am absolutely the strongest on POSIX/UNIX-family systems. But that doesn’t stop me from being able to learn it and handle it. Even if it does take longer.

The difference between me and the unidentified person in my last post? I’ll spend any resources necessary—time, money, effort—to learn what I need to learn to get the job done. I don’t cut corners. My goal isn’t to get everything done sloppy and fast. Even if it takes me longer, I’d rather know and understand the problem—and its solution!—completely before moving forward with doing anything about it. Especially if I can find a short-term workaround that will enable me to come up with a quality solution. I eschew willful ignorance in my field. Do you?

One of those situations where it is really annoying is when trying to do things that ought to be really simple. For example, when writing an accessor function for a boolean value, one would expect to be able to just do it, right? But because PHP isn’t anywhere close to a strongly, statically typed language, this is impossible to do simply. The programmer must be burdened with checking the types, or hope that the user will never make a mistake.

Some days I wonder why I’m not writing Web application software in C using SCGI…

We pay cash at restaurants especially ones of certain
nationalities.

The context of this quote is a discussion on credit/debit card usage, and this statement came at the tail end of how care must be taken to ensure that one is not subjected to fraudulent charges (nevermind the fact that banks in the U.S. mostly do zero-liability these days). This has spawned a rather heated discussion, which apparently resulted in the person who made that comment leaving the mailing list.  The whole idea that we continue to change our behavior depending on the ethnicity of the person(s) we are around is nothing short of infuriating. It is like we fail to understand that qualities like honesty and trustworthiness are markers of an individual.

What does the above statement say? It says that the poster of that statement feels that he cannot trust “certain nationalities”. This person later made the claim that it was sad that people could find racism in nearly any remark—but the thing is, it is right there, not even hidden from view. It’s blatant.

Even worse, when the poster of the comment above was called out on it, most of the people on the mailing list jumped to this person’s aid to defend them. Honestly, I do not know what is worse: the fact that this person said this in the first place, or the fact that the majority of the mailing list’s members rallied up on the side of that person. Quite possibly, I think the latter, because that shows that we still have in society the notion that racism is somehow acceptable, and that the more veiled or subtle it is, the more okay and polite it is. I find that downright offensive.

It is certainly enough to make me consider dropping the mailing list and participation in the group altogether. It is difficult to be in a group when you cannot even look at its members—your peers—with respect, even if they are some of the smartest minds you know in a particular field. Why be part of it at all?

So, this is a pretty simple-sounding network, yes? It should be—it is just a handful of computer systems. However, there is a problem—a pretty large one, I think. There is very little in the way of either policy or convention on the network. Some users use their documents directory for storing their documents, others store their documents on their desktop, others somewhere differently altogether on the C drive, others on a network share in a public storage space. There are different versions of software on the various machines, more-or-less updated when someone thinks about it, I think.

This is a perfect scenario which shows why a business network should be centrally managed in some form. Note that I am not saying that each machine should be a bit-for-bit mirror image of the one next to it, though that is certainly a possibility. I think that people should be able to use their own choice of things like email client or Web browser software, because everyone is different. But when you have different client applications fulfilling a role on the individual workstations, you have to take a centralized approach to ensuring that things like the email is all backed up.

Furthermore, if you don’t take a centralized approach to backing up such data, it is very difficult to centralize the network storage. Think about adding a domain controller (that is Windows speak for a central server which handles authentication and authorization, as well as file and printer sharing and things like roaming profiles) to such a network. I expect that with multi-gigabyte mail files, things will be very slow at first—and that likely the only fix for them that is going to be viable in the long term is to centralize more infrastructure.

I am too tired to expand more on my thoughts on what I have learned and where it is heading, but the Reader’s Digest version of the point reads something like this: If you are a small to medium sized business, make sure that you have someone who is competent in both system and network administration, and make sure that they are a part of your business from day one. Like writing software, building up a technical infrastructure without careful thought and design is hazardous and comes with many hidden and unpredictable costs. It is best to head those things off right from the start; to delay only amplifies the cost of fixing the underlying problems and puts oneself in the position where fixing one issue can have a domino-like effect and create more new problems.

For my current situation, I think I am going to have to seriously re-think how this whole setup is done. What I do not yet know is how to quickly and efficiently bring things into shape. A bit of training and education may be required, and certainly the removal of a lot of unnecessarily-granted privileges on the workstations. That, too, should be something caught early-on: do not let every person in a business run with administrator privilege, unless they are an administrator (and even they should only run with administrator privilege when they are actually doing something that requires that privilege). If everyone is an administrator, there is little to no control on how things are done in a network, and it can get messy.

I have a lot more reading to do, as well.

Well, anyway, it is way past my bedtime.  Time for sleep.

The Free Software Foundation objected to UDI for various reasons. Mostly, I think, it was because they were afraid that people who are not them would choose to use drivers that were non-free. As I’ve written about before here, there are people who think that forcing people to use free software is somehow freedom—and I will not go into it in any great depth here, because I have done that in the past. Suffice it to say that forcing anything is not freedom; it cannot be freedom. So, the Free Software Foundation, I think, was really afraid that they would have to do more work to be able to stick to their own requirement of using 100% free software on their own computer systems. (And hey, Roy, if you’re reading—I’m not saying that the FSF is wrong, and I’m not putting myself in a position opposite of that of the FSF. I suspect you think so anyway, but hey, I just figured I would point that out.)

Even if the free software operating systems did not adopt the UDI specification, why didn’t proprietary operating systems? This is perhaps the most puzzling thing to me. It seems that in this event, none of the operating systems—free or proprietary—did what would have made sense. After all, even if only Apple and Microsoft adopted a common device driver specification, that would save a lot of time, effort, and improve user experience all the way around. Apple users would be able to use all the hardware that Microsoft users could use—and the inverse would also be true. The amount of time that device driver authors would have to spend writing and debugging driver code would go way down—free software driver authors would be able to write a driver once, for example, and all systems (including free software systems that chose to support the specification) would benefit.

I could see an objection of a driver specification that was binary-only. However, UDI was not—it mandated an ABI so that drivers that are built for a particular platform were binary-compatible with operating systems on the same platform, but it also mandated an API, so that drivers would be source compatible to any operating system that implemented the specification, on any platform. That by itself would seem to me to be positive motivation to hardware manufacturers to release the source code to drivers so that they can support operating systems that are on platforms that do not exist yet, or have not been considered (or have been considered to be nonviable or unsupported platforms).

So, I have to wonder why a common device driver specification was never implemented in various operating systems. It would seem to be a common sense thing, especially given that there are so many operating systems. It would make the coexistence of operating systems a lot easier, and it would promote choice. It might encourage bits of proprietary code on free software operating systems, but it would also enable people to drop the excuse that “free operating system x does not support device y”, and would as a result potentially increase the number of free software programs and operating systems in use, even if there is a minor cost in terms of certain drivers. And those drivers could always be replaced—a common driver specification would make it easier to understand the structure of drivers generally, and make it easier for lawful, clean-room reverse engineering to be done on those drivers.

Imagine, for example, if drivers for graphics cards, TV tuner cards, video and audio encoding/decoding cards, modems, storage chipsets, motherboard chipsets, USB chipsets, IEEE-1394 chipsets, graphics tablet devices, touch screens, debugging interfaces, network devices, and so forth were all written to a common specification, it would reduce the amount of code which needed testing. It would increase user choice in both hardware and operating systems—something which I still hold is quite likely the most valuable freedom we have. It would increase reliability, since the users of Windows, OS X, Linux, the various BSD systems, and other, not-so-mainstream operating systems would be able to run the same driver code and collectively supply debugging information and perform testing in a multitude of environments. It would increase security, because then common code that is well-known could be used on all platforms and not just the one it was written for. It would do for device drivers what POSIX has done for user-mode application software. I do not believe that I could be convinced that this would be anything other than a good thing.

Also, it could bring back old operating systems.  Imagine what life could be like, for example, if OS/2 had a “UDI driver” written for it, and it could then take advantage of newer drivers never intended for it. Or any other very old operating system which is no longer supported and could still be useful, for any of a number of reasons…

“You can have freedom without choice.”

That someone could even come up with this one is just amazing to me. Note that this is not an exact quote, but it is the summary of Friday’s topic. For example, this summary comes from the idea that Canonical is bad for considering making mainstream non-free software available for Ubuntu based on user preferences. It does not matter who came up with it, of course, but the important thing is that it be called what it is: patently absurd. The ability to choose is a major part of what freedom—or liberty—is. If you cannot make a choice on a matter, then by definition you do not have freedom in the context of that matter. It is quite simple and self-explanatory. Canonical is seeking to increase freedom here, not take it away. Some people actually want to use non-free software; others may not want to use it, but aren’t aware of alternatives. The latter group of people should have our focus with regard to education (but then we should let them make the choice for themselves!).

Note that I am not one of these people: I would rather use free software because of the liberty it gives me that I have come to expect over the years. But I am not going to tell someone else that they are harming me because they would rather use non-free software that is familiar to them. All I can do is show them that there are free alternatives that exist. I cannot—and I will not—make them use it or make them feel bad for not using it. I may not like proprietary software for a variety of reasons, but I will defend people’s right to use it just as I will defend even a stupid person’s right to spew nonsense by way of speech or written word. In other words, “I disapprove of what you say, but I will defend to the death your right to say it,” or perhaps more appropriately, “I [may] disapprove of what [software you run], but I will defend to the death your right to [run] it.” Even I use a package or two that is proprietary in nature (though it is looking like I will not have to do so for much longer, given the efforts to replace these packages with equivalent free software).

It is worth it to note that by adding non-free software to Ubuntu, the free software that is already there does not change. The mere existence of non-free software within its repositories does not make Ubuntu somehow bad or evil. It would add choices that do not currently exist, and that one such as myself or yourself can certainly opt out of—I most likely would, for the most part, as I do not need to depend on non-free application software, and I only use non-free drivers if I have hardware where anything else is nonviable (and only until there are functional free software drivers). Did you know that Ubuntu has an option in the installer to only install free software? Can you say that for your favorite desktop operating system distribution, whatever that might be?

The response to this idea, then, is that without choice, there is very little—if any, really—freedom. The thing that gives us freedom with free software is that we are able to to download the source code, to review/audit it, to change it to fit our needs or fix a problem, and to share those changes. If we cannot do those things, then it is not free software; see the essential freedoms. But non-free software inside a distribution is not something that should not cause you great consternation even if you are among the most dedicated of freedom advocates, for if you are a true advocate of freedom then by definition you must respect a computer user’s freedom of choice. Remember that we choose to run free software because of the benefits it brings to us; we choose to improve upon free software for much the same reason. Eventually, I think that free software will once again become the norm for computer software, on merit alone, for no other reason than the development, release, and usage of free software is a highly practical solution for many things ranging from library code to application software to complete operating systems. It is worth noting that free content—which is similar in concept to free software, which itself is merely a specific application of freedom itself—also appears to making major headway towards becoming mainstream; it is doing so more quickly than free software is, but there is every reason to believe that free software will follow, for it is already.

An Example

Imagine that you are in a store, because you need some milk for dinner some night. You always get 1 gallon of 2%. But, the store has stopped carrying it, because more people buy whole milk and they were throwing away the 2% milk—demand was low, supply got to be too high, so they just stopped carrying it altogether. You leave the store and head to the next in the same town and you find the same thing there. You have a choice of stores to go to, and you have made the choice to go buy yourself some milk. But there is only one type of milk. You no longer have the choice to buy 2% where you are, and so effectively, your freedom to buy it has been taken away. (Of course, you can make 2% milk from whole milk (and make whole from 2% even, or even butter), but I suspect just as many people want to do that as want to write their own free software that they demand simply must exist, but doesn’t yet).

Now, the point here is that there is more than one freedom in play: the freedom of the store to stock (or not stock) various products, which affects your freedom as a consumer to buy the product you want. In the case of software, and choice, if the software you are running gives you all the choices you want, then it fits your needs. If it does not, then you are not going to be able to use it the way you want. Now you have two choices: you can do the work that it would take to make your desired choice possible, or you can use another system (free or proprietary) that will give you the choice that you want. Many people will choose the latter, especially if they are non-programmers. Though I’ve seen programmers also choose to use proprietary systems for something that they could themselves implement. That is their choice, of course. After all, if you really wanted 2% milk, you would have the same choice: make it yourself, or drive to the next town over which might have it available for you (assuming that there is some in stock and that the stores neighboring towns have not also decided to stop stocking 2% milk).

Ubuntu One: The Reason Behind This

This discussion came up because someone on identi.ca made the claim that Canonical is forcing proprietary software into Ubuntu by way of the Ubuntu One client software. I cannot even begin to state just how woefully incorrect this point of view is. First off: the only thing added to Ubuntu is the ability to connect to Ubuntu One, and the software that was added to Ubuntu do to that is licensed under Version 3 of the GNU General Public License. The claim made in response to that was that Ubuntu One is only partly free software, because the server is somewhere else and has not been released. As we shall soon see, that claim is nonsensical—it depends on an extremely naïve view of how software actually works in order to make sense, really.

So, first things first: Ubuntu One, which was added to Ubuntu 9.04, is not proprietary software. The proof rests in the fact that it GNU GPL v3.0, and we know a priori that software licensed under the GPL is free software, so we do not need to go further on that point.

Now, because the software in question added to Ubuntu is free software, we can read it. The essential freedoms granted to us by truly free software ensure this, and the GPLv3 is indeed a truly free software license because it grants those freedoms. Because we are able to study the software and see how it communicates with the server. Once we know how to communicate with the server, we can write that up and design a server that communicates exactly the same way. From there, it is just a matter of patching the sync dæmon that is in Ubuntu to talk to an arbitrary, Ubuntu One compatible server. To determine how to do that, one need only read the Python source code contained in the python-ubuntuone-storageprotocol and python-ubuntuone-client packages. If you do not know Python well, you might expect to spend several days doing that, but if it bothers you so tremendously that you are going to practically start a flame war over it, you may find it worth it to do so.

Of course, the other side to that is this: if you really want Ubuntu One to talk to an arbitrary server that runs free software, and you want that free software to be written, you can fund the effort to write the free software. Approach a proficient developer somewhere out there on the Internet and ask them how much they’d charge to write a server for Ubuntu One. You might not be able to afford the fund the project entirely, but if you get a number from someone, you can start a coordinated effort to raise the funds. If you are lucky enough to be able to fund the whole project, then do so: it is but one way that you can help provide something back to the community. This does not apply to just an implementation of the Ubuntu One protocol, it could apply to anything that you see that is missing and needs to be created. Or you could spend time learning what you need to learn to pick up the project yourself, if you care for the project that deeply. The most important attribute that a person can have in order to get started with development is motivation—James Westby reminded me of this a couple of years ago, something which I had forgotten.

Perceptions: Another (Possible) Reason

It was suggested to me that another possible reason that people would object to having non-free software inside an operating system distribution such as Ubuntu is that they are afraid that the proprietary options have higher quality, or offer superior features, or provide functionality that is not offered by any existing free software. Thus, they have this perception that by adding such non-free software into a distribution like Ubuntu, people will automatically use and prefer it over free software. This simply is not the case. Sure, some people will use iTunes if it is available on Ubuntu. Maybe many people would. I might even do so, if it were legally available for me to use that way and if it supports the purchase of DRM-free music. However, if there were a free software client for the iTunes store, I’d much prefer to use that. To my knowledge, however, there is no such thing that exists.

If there is not a free software alternative for a non-free component inside a distribution of software, if you are offended by that, then by all means, create a free software alternative for it! As mentioned above, you can start on such a project’s development, or you can look for people that would be interested in volunteering for it and coordinating them, or you can put up funds to pay developers to implement it. If you have money, this can be the easy part: find someone who is willing to accept payment for the service of implementing the free software alternative for whatever it is that someone else has funded, wrote, and released as proprietary software. It is not like free software is developed without cost (and if you think that it is, then you seriously do not understand what free software is or anything about the world of free software and have no standing to be getting mad when a company spends money writing software and does not release it as free software. You can try to write companies that write such software and ask them if they will give you any form of written specifications for the software, or an interface definition, or something along those lines. The worst thing that could happen is that you will be told “no”. And do so nicely, or they’ll be more inclined to tell you “bugger off” instead of simply “no”.

“Allowing users to choose proprietary software is anti-freedom.”

Nothing could be farther from the truth; it is the same, in fact, as the above statement that one can have freedom without choice. For example, if Ubuntu adopts iTunes and makes it so that you can “sudo aptitude install itunes” in the future, that is not a bad thing! How can it be—It contributes to the ability to choose, and thereby contributes to the freedom of the end-user. If you are a die-hard free software supporter and do not want to run non-free software on your system, then there is a very simple solution for you: simply don’t install it.  That is a valid solution to the problem. There are tools already available that can be run as a cron job and report on any non-free software that you might have accidentally (or even intentionally) installed. If you are worried about additional non-free software getting into Ubuntu, then help enhance those tools. Or write a GUI front-end for something like the virtual RMS program and work to get that included into Ubuntu as well, perhaps something that can run every time you login to the computer, or that runs as a persistent process that watches the package database on your distribution of choice for updates and then checks to see if newly installed software is non-free and alerts the user. Of course, it’d be most effective as an opt-in system, and not an opt-out one where it would just be annoying.

There is no way, then, that freedom is actually reduced in this way when another choice becomes available. If iTunes were to be included in the repositories (and I suspect it would be, like the restricted, universe and multiverse repositories, a separate opt-in repository; perhaps simply “proprietary” would be fitting), this does not reduce your ability to choose to run a free software media player and manager like Banshee, or Rhythmbox, or even AmaroK if you are so inclined to run that KDE stuff.

Once upon a time, FUD (fear, uncertainty, and doubt) was the tool of Microsoft. We (the free software world) completely hated it when Microsoft would put out FUD, because we would then have to fight that FUD by way of explanation and demonstration. Well, some time ago, a subgroup of the free software world decided to start using FUD themselves—it was done with Mono, and it is being done now with just a survey asking people what sort of software they would like to see in Ubuntu. Now, those of us who are left who are advocates of liberty—both personal and societal—are stuck potentially fighting two battles. One with Microsoft’s FUD—such as the constant notion that you have to pay for software—and one with the “free software evangelists” FUD, who have even gone so far as to say that people should not use certain types of free software (the one who calls himself “The Open Sourcer” even still today tells people to remove certain truly free software from their systems). The truth is somewhere in the middle, between these two ends of the spectrum.

Conclusion

Back to the point at hand: to say that giving a person a choice is a constraint on that person’s freedom, that is doublespeak.; it is saying that “slavery is freedom,” albeit to a lesser degree than that very melodramatic extreme—it simply does not make sense. The concept just does not make sense unless the words that are used to express the concept are dramatically redefined to mean things vastly different from what standard English dictionaries define them to be. The only reason that one has to try to convince someone that additional choice is a constraint on freedom is to try to convince people of things that are not true; to install fear, uncertainty, and doubt into people. This is the sort of behavior that—no matter what community it originates from—is completely immoral, unethical, and absolutely unacceptable. It’s dishonest, and for those of you who know me personally, you know what I think of dishonesty.

Through something of a comedy of errors, the server suffered some strange software issues that prevented it from working this weekend.  There was a bug in a recent update to the server software (running testing software) and that caused longer downtime than it should have due to various interactions between things on the server.  The good news is that this is mostly fixed.

Additionally, this downtime has taught me that there is yet still more to do in terms of getting the server able to stand back up on its own.  I’ve simplified the server’s setup a bit, and I have to write some scripts and other little glue here and there to tie down some of the things I’m doing so that the server can do things like go down and come back up without issues, all by itself.  Getting that done would be generally a good thing.  First things first, I have to figure out a decently reliable way to shutdown the system without having to do something like kill the containers and not give them the chance to cleanly shut down.  Ideally, there would be some sort of command that could be run on the system that would enable the containers to be shutdown.  This is slightly challenging, because you cannot just chroot into the directory tree that the VMs are running in and kill processes, because things like /proc inside the container aren’t visible to tools running on the host.  Oops.

So, I have some work left yet in terms of getting the server going the way it needs to be again.  After that, I’ll be working on it over the next couple of weekends to try to increase its robustness, and so planned weekend downtime for *.trausch.us and www.mischiefinoverdrive.us can be expected.