There are a number of things that I did not address in my post yesterday. My only goal in that post was to illustrate a (rather simple) network that had four subnets. I probably should have posted this article first, but I did not really think about that. This post is going to be somewhat long and very link-heavy. I don’t expect that you’ll be able to make it through this post (and all it references) in a single reading session. It’s more like a crash-course in all that which is computer networking from the bottom up, to a certain extent. This post, I have decided, is going to be the first post in a series, working toward the goal of building a basic understanding of the underpinnings of computer networks up to and including both IPv4 and IPv6.

Before you get too deep in here, note that you’re likely to encounter things that you know already in it. But, it should be useful for anyone attempting to learn networking from the ground up. Most people in IT who are relatively new to it (or have been shoehorned into a position of supporting a network) have only really learned about bits and pieces of it, often as a result of being practical about learning only what is needed to get things done. As a result, I don’t really know how much “below” IP people who read this post know about. Even if you know about all of this stuff, I’d appreciate a full proofreading.

All of the IP addresses that you will find in this post are pulled out of pools reserved for documentation and examples. None of the IP addresses you find in this post will work on the public Internet. It might be possible to use them in isolated network islands, but there is network space reserved for use on islands; see RFC 1918 for IPv4 and RFC 4193 for IPv6 for address space reserved by IANA for such purposes. The MAC addresses in this post technically come from Xerox’s pool of MAC addresses. Don’t use them anywhere. I could not find a MAC address prefix dedicated for documentation/example purposes. If someone happens to know of just that, please let me know.

There are two network models: the four layer IP model, (I am calling it the IP model and not the TCP/IP model because I won’t carry forward the misnomer “TCP/IP”, as TCP is far from the only thing that runs on IP) and the seven layer OSI model. I use the seven layer OSI model in part, because it is the model that I was taught when I learned about networking, but also in part because it is more descriptive and has a finer level of granularity—it is easier to talk about more layers that are smaller in scope than fewer layers that are larger in scope—at least, IMHO.

The OSI Network Model

The OSI model (Open Systems Interconnection model)—sometimes called the ISO network model since it was defined as an ISO standard—is the basis for the discussion that I will use in this post (indeed, it is the basis that I use for all of my discussions on networking). The OSI model has been around for a very long time, dating back to at least 1980 and probably even further. I wouldn’t know without a lot more digging, since I wasn’t around when it was created.

The OSI model has seven layers. Conceptually, each layer only communicates with the one layer above and the one layer below it (with exception for layer 1, which has no layer below it, and layer 7, which is the top). The seven layers of the OSI model are:

1. Physical layer. In your average Ethernet network, this is essentially the cable and the transceivers at either end of the cable. For dial-up networking, it consists of the serial lines at both ends, the modems at both ends, and the telephone network in between. This layer of the OSI model is concerned only with the movement of bits across the physical medium and passing data to and from layer 2. This is the hardware-specific layer and it is spoken of in terms of electrical connections, circuits, and signals.
2. Data Link Layer. In your average Ethernet network, this is the frame-based protocol (also called Ethernet!) that is used. For dial-up networking and networking on cell phones, it is (most often) PPP. This layer receives bits from layer 1, interprets those bits as frames, strips the framing off, and sends the frame’s payload data up to layer 3. The inverse operation is performed when data is received from layer 3: framing is added and the bits are sent down to layer 1.
3. Network Layer. No longer do we care about what type of physical network is in use: layer 3 is where IPv4, IPv6, ICMP, ICMPv6, IGMP, IPX, AppleTalk, and so forth all live. The network layer is responsible for packet routing, filtering, and delivery. It receives data from layer 2, strips the network protocol framing off, and sends the data up to layer 4; the inverse is taking data from layer 4, adding network framing, and passing the packet down to layer 2.
4. Transport Layer. This is where TCP, UDP, SCTP, and others live. The transport protocol determines whether the transport is connection-oriented or not, whether it sends streams or datagrams (or can do either or both), whether or not reliability is provided, and so forth. This is the layer used by application software using the Berkeley sockets API when talking directly to a socket (in truth, the BSD sockets API also exposes a bit of the Network Layer, so it’s not a pure Transport Layer wrapper).
5. Session Layer. This layer is not often discussed in relation to IP networks, because it considers the duties of layer 5 to be part of its application layer. However, the presence of layer 5 is useful as an abstraction when talking about sessions and how they can be thought of, and libraries that provide a sessions support mechanism on top of the BSD sockets API effectively implement the OSI Session Layer for applications that talk over IP. NetBIOS and L2TP are two such protocols which could be considered to be in this layer (and L2TP is an example of “a network in a network” in terms of the OSI network model, but we will skip discussion of recursive networking for later: even though we use it all the time in IT, most people don’t realize it and think the concept is more confusing than the application).
6. Presentation Layer. Like the Session Layer, the TCP/IP model considers the duties of layer 6 to be part of its application layer. MIME, SSL, and TLS can all be considered to be OSI Presentation Layer components. The OpenSSL library effectively implements SSL (and TLS) as a layer by wrapping the BSD sockets API.
7. Application Layer. This is the layer concerned with actual applications: DHCP, DNS, FTP, HTTP, POP3, IMAP, IRC, XMPP, Telnet, SSH… the list goes on and on. The application layer represents application protocols that we use every day for IP address provisioning, turning names into IP addresses, transferring files, browsing the Web, email, chat, remote control of computers, and so forth. Many of the things that we consider application protocols in the context of an IP network also implement things elsewhere: for example, SSH is an application protocol in IP networking that performs tasks at OSI layers 5, 6, and 7.

As I mention in layers 5, 6, and 7 in that list: the IP model of networking essentially merges those three layers together into a single layer. It also merges layer 1 and 2 into a single layer. Because of that, it is harder to focus on a single precise task in the IP model. However, it is worth mentioning that even in the IP model, the Internet Layer (equivalent to OSI’s Network Layer) and the Transport Layer (equivalent to OSI’s Transport Layer) are separate things.

As you can imagine, the layer looks like a stack. This is probably where the term “network stack” comes from; software implementations often use the layer model, resulting in a suite of software “stacked” one atop another. In theory any one layer’s implementation can be altered or completely changed, and as long as it preserves the interface it has with the one layer below it and the one layer above it, nobody should really notice. In practice, it works almost as smoothly: applications that use TCP for transport are able to work on IPv6 as well, usually with few changes to the source code in order to select an IP-version agnostic socket type (if, that is, any changes are required at all). This is also why IPv6 support is nearly ubiquitous (at least, in popularly used free software).

An Overview of OSI Layers 1 and 2

Before we can continue on to discussion of the many things that IP does for us, it would be helpful to take a closer look at the OSI Physical and Data Link layers. These are the layers upon which both IPv4 and IPv6 build in order to provide us with the convenient networking that we are so used to today. We will also take a look at some of the network equipment that operates at these layers, so that we can understand how they fit into our networks. For the purposes of this section, let us imagine two computers (we will call them A and B) which use a standard gigabit Ethernet (1000BASE-T) to connect to each other and provide a network. As we are limiting our discussion to OSI layers 1 and 2, in this example we will not care what is handling the jobs of layers 3 through 7.

The Physical Layer provides the medium upon which communication can take place, as well as the electronics which directly use the medium. So, if we look at computer A, we have the following things that represent the Physical Layer:

• A Category 5 cable. The cable is the actual transmission medium: data “flows” over the cable.
• The Ethernet port, (an 8P8C modular connector) which the cable is plugged into. The wires in the cable connect with wires in the port, which run to the Ethernet interface’s transceiver.
• The Ethernet interface’s transceiver, which is responsible for transmitting (and receiving) binary digits (bits) to and from the wires in the cable.

Operating systems are not concerned about the physical layer in a computer network; that is a job for hardware to handle all on its own. (This is not the case if the Ethernet interface offloads its processing to the host system’s CPU, but we will disregard this as an unnecessary complexity for the time being.) In this specific example, the role of the Physical Layer is to receive data from the Data Link Layer, serialize that data into bits, and transmit it over the cable. Likewise, bits that it receives from the cable are deserialized and sent up to the Data Link Layer. At the Physical Layer, the only thing defined is the connection between two (or more) nodes on a network segment as a bus or star network. Some examples of layer 1 network devices are ports, connectors, cables, terminators, and hubs (despite the link being to “Ethernet hub”, other types of networks can have hubs, too; they’re not specific to Ethernet networks). A hub is just a device that has more than two ports for network devices to plug into, and everything that is transmitted on the hub is sent to all of the ports at once; effectively, a hub is a repeater and connects all of the nodes together as if they were all sharing a single cable, but they are not. Bridges and switches, two types of popular networking equipment, are not OSI layer 1 devices as we will soon see.

The Data Link Layer (or, layer 2 in the OSI model) can be thought of as providing two “sublayers”: Logical Link Control (LLC) and Media Access Control (MAC). The term MAC address refers to the address that a particular piece of network equipment uses at the MAC sublayer of the Data Link Layer. LLC provides the ability to multiplex many different network protocols over the same physical medium, while MAC provides an address space as well as the rules by which transmission and reception over a shared medium work. Think about a network segment as a town, and a MAC address as a street address within that town. It’s possible to have multiple street addresses that are identical as long as they are in different towns, but if two places have an identical address in the same town there are problems such as incorrectly delivered or lost mail. Instead of lost mail, a duplicate MAC address on a single network results in frames not being delivered, being delivered twice, or being delivered sometimes to one or the other of the nodes in conflict. The behavior in such a situation can be extremely erratic. It is only a common problem in environments where humans set MAC addresses, such as when using certain types of virtualization software.

The Data Link Layer is the layer that handles addressing, framing, and sharing of the network segment. It is still not possible to do true internetwork routing using just the Physical and the Data Link layers. You can, however, do two new things at layer 2: network bridging, and packet switching. In fact, every network switch is logically just a multiport network bridge where all of the ports are bridged together—it is important to realize that. This brings with it two benefits: more collision domains (and thus ability to increase the usable bandwidth on a network segment given certain types of traffic patterns) and the ability to join multiple physical network segments together into a single logical network segment. Also, despite a switch (or a bridge) being a layer 2 networking device, they are invisible to the networks. A bridge behaves as an intelligent, transparent proxy between two networks, turning two physical Ethernet segments into a single logical Ethernet segment; a switch is nothing more than a bridge with multiple ports. Yes, that does mean that if you have an 8-port switch—and you’re using all of them—what you have is 8 distinct physical Ethernet segments, but it behaves as if it were a single Ethernet segment, so it is a single logical Ethernet segment.

On today’s networks it is very rare to not have a switch somewhere, and while bridges are not exactly commonplace they are used quite a bit. Bridges and switches both make it much easier to manage a large network as if it were a single large network, but without many of the problems that large networks would generate without them. Prior to the widespread availability of inexpensive bridges and switches the two primary networking tools were hubs and routers. Networks were forced to be compartmentalized in order to keep the size of its collision domains manageable. There were also limitations in the length of cable that could be present between two nodes that wanted to communicate with each other.

Switches do not entirely eliminate all of these pains, though. There is a limit to the length of a cable in an Ethernet network, and there is a limit to how many switches may be present between two systems who wish to talk to each other on a logical subnet. If you find that you need to exceed those limitations, your only option is to use subnetworks and add a router to the setup.

Enough about the Physical and Data Link Layers! Back to the Computers!

At the beginning of the last section I mentioned two computers: A and B. We now have enough information to talk about how they communicate with each other. Computer A would like to say “Hello!” to computer B. Computer A needs to know computer B’s address on the network (or needs to know how to broadcast over the network) in order to do so. For now, we’ll just assume that we want to broadcast—it makes things more simple! In terms of the Data Link Layer, the broadcast MAC address is FF:FF:FF:FF:FF:FF, or the “all F” or “all ones” address. Any layer 2 packet sent to this address is received by all systems on the network (though it may not be processed by all of them). Let us assume that computer B is listening for broadcast messages, and whenever it sees “Hello!” sent to the broadcast address, it says “Hi there!” back, also sending to the broadcast MAC address on the network.

Now, let’s say that computer A has the MAC address 00:00:01:00:00:01 and that computer B has the MAC address 00:00:01:00:00:02 (no, that would almost certainly not happen on a real network). Computer A knows that it wants to address computer B with “Hello!”, so it sends “Hello!” to address 00:00:01:00:00:02. The frame is transmitted on the wire, computer B’s network interface sees the frame and because of the address decides to read and process it. It then generates a new frame, addressed back to 00:00:01:00:00:01, which says “Hi there!”, and the exchange is complete.

It is important to keep in mind that there is sufficient power at this layer to perform only simple networking: unicast or broadcast on a single network segment are the only possibilities. Therefore, it is not suitable for internetworking by itself. In order to achieve internetworking, we have to have the ability for computer A to talk to computer C, even when computer C is not on the local link, perhaps on the other side of computer B (in which case, computer B would be a router). However, a router is not a layer 2 concept: it is a layer 3 concept. And so, guess what we’re going to talk about now?

OSI Layer 3: The Network Layer

It is at this layer that it becomes possible to perform internetworking. That is what makes the IPv4 and IPv6 protocols so special: they enable the ability to have a network so large that no one can completely imagine it.  A network is composed of smaller networks that participate in providing transit for it, which consist of even smaller networks, which themselves may have networks (and this recursion can be nested arbitrarily deep; there is no limit on the depth of the tree that is the Internet). [question: would a diagram here showing the structure as a tree data structure be useful here?] This means that you may build networks consisting of different physical network types, and they can still be linked together and participate as a single, harmonious (we hope) network.

In fact, this is precisely how you are able to be on the Internet reading this at all. Your computer has either an IPv4 or an IPv6 address on the public Internet, or it has a private IPv4 address on a network that is essentially transparently proxied to the Internet. Either way, your computer was able to reach my computer, and request that this blog post be transferred to you in order for you to display it. When I think about all of the things that make this wonderfully huge network actually work, it is hard to imagine—even though I have been on the Internet for roughly two-thirds of my life.

Let us now expand our example network from earlier. Imagine computer A has two Ethernet ports. A cable is plugged into the first Ethernet port, and the other end of that cable is plugged into computer B. Yet another cable is plugged into A’s second Ethernet port, and is plugged into computer C. If you visualize the three computer systems, there is A, which is the vertex of two lines that are the cables in its two Ethernet ports, and computers B and C are at the ends of those two lines (said perhaps more simply: if B and C had a cable connecting the two to each other, we would have a triangle). We still have no connection to any other computer network, including the Internet, in this example.

At this point, what we want is for computer B to be able to say “Hello!” to computer C. But we have an obstacle in the way: there are two network cards in computer A, and B is attached to one while C is attached to the other one. What we have is two network segments, and all of the following statements are true:

1. Computer B is on one network segment, connected to A.
2. Computer C is on one network segment, also connected to A.
3. Computer A is on two network segments, one of each is connected to computers B and C.

Since we are string for an example in routing, computer B and computer C will be on different IP networks; computer A will be on both of these networks. This means that:

1. The first Ethernet interface on computer A (which we will call eth0) will have the IP address 203.0.113.1/29.
2. The second Ethernet interface on computer A (which we will call eth1) will have the IP address 203.0.113.9/29.
3. Computer B’s eth0 will have the IP address 203.0.113.2/29 and it will be configured with a default gateway of 203.0.113.1.
4. Computer C’s eth0 will have the IP address 203.0.113.10/29 and it will be configured with a default gateway of 203.0.113.9.

We could say that we have two sibling networks. That is, we have two networks, neither of which are a physical subnetwork of the other one. Since we are disconnected from any other networks, this addressing scheme works just dandy for us. And, with a suitably sane operating system, we have nothing more to do, either: all three of these computer systems ought to be able to talk to each other just fine now, because:

1. When we added the IP address 203.0.113.1/29 to eth0 on computer A, it should have put an entry in its local routing table that 203.0.113.0/29 was reachable via eth0.
2. When we added the IP address 203.0.113.9/29 to eth1 on computer A, it should have put an entry in its local routing table that 203.0.113.8/29 was reachable via eth1.
3. Computer B was configured with an explicit default gateway, and that gateway is set to point at computer A.
4. Computer C was configured with an explicit default gateway, and that gateway is set to point at computer A.

There is one catch: the Linux kernel will not behave as a router if it has not been configured to do so. That is, it won’t forward packets arriving out from eth0 but obviously destined to eth1 to eth1; instead it must be told to do so. I suspect that other operating systems have similar settings that must be flipped in order for the system to become a router. Once that is enabled, the routing table is all that is necessary in order for the kernel to make routing decisions. An added bonus is that if the kernel receives a packet destined for a network that it knows it cannot connect to, it will send an ICMP reply indicating that the destination network is unreachable (e.g., there is no route to the requested node). So, if you’re trying this out at home, you may have to enable some form of IP forwarding for things to work, but they’ll work. Computer A will behave as a router.

In terms of network diagnostic tools, B should be able to ping both A and C; C should be able to ping both B and A; and a traceroute between B and C should show A as the first hop and C as the second hop. We have a functional, routed, network! However, this also means that network broadcasts on computer B’s network will not reach computer C’s network (and vice versa). That is to say, computer C is outside of computer B’s broadcast domain: they can both broadcast such that computer A will see the broadcast on its respective interface, but they cannot broadcast to each other. (There is something called multicast, and if computer A were configured as a multicast router, systems on both networks connected to A could subscribe to multicast groups managed by A and thus could send things to the multicast group and computer/router A would pass them along to the other network. However, that’s pretty advanced and we’re going to ignore that for now.)

Great, but how does all that work anyway?

So we know now that computer B can talk to computer C—even though there is no direct physical connection between computers B and C. We know that this is done because A is set up to route packets between B and C. So, how does an Ethernet packet from B wind up getting to C? The answer is that it does not. Huh?

Right. It does not. Because routing is a layer 3 thing, it is layer 3′s job to figure out what the next hop is for the IP packet. If computer B is attempting to send to computer C, it knows that it cannot do so directly. Computer B knows this because it is only attached to the 203.0.113.0/29 network, and 209.0.113.10 is not in 203.0.113.0/29. So it looks to its default route (that is, the system specified as the default gateway) and figures out what the next hop’s IP address is.

Here, we are going to stop for a minute and figure out how layer 3 gets anything done. When two computers are on the same IPv4 subnet it is simple: if computer A wants to send to computer B, computer A makes an Address Resolution Protocol (ARP) request; this is a broadcast, asking the network “who has the IP address 203.0.113.2?” and expecting a response. It will try this a few times before giving up; if it does give up, then the host is deemed to be unreachable: either no systems on the network have that address, or something is malfunctioning and it is not receiving, not transmitting, or otherwise impaired. (An aside: this is one thing that works differently in IPv6; instead of using ARP, IPv6 uses Neighbor Discovery Protocol (NDP) to find a host on the local network.)

So, when computer B attempts to talk to computer C, here is what happens:

1. Computer B notices that 203.0.113.10 is not in its own network (which is 203.0.113.0/29).
2. Computer B looks in its routing table, and it sees that it does not have a route to a network that contains 203.0.113.10.
3. Computer B therefore looks again in its routing table to see if it has a default route. It does.
4. Computer B looks up the address for the gateway specified in the default route (203.0.113.1).
5. Computer B makes an ARP request to get the MAC address for the node with the IP address 203.0.113.1.
6. Computer A, having the IP address 203.0.113.1, sees this ARP request and says “I have that IP address.”
7. Computer B sees the reply from computer A, and makes a note of its MAC address.
8. Now, computer B takes the IP frame as a sequence of bytes and sends it down to layer 2, and tells layer 2, “Give this packet to this MAC address,” and layer 2 does its thing. The packet makes its way to the wire destined for the MAC address of the interface eth0 in computer A.
9. Computer A receives the layer 2 frame, reads the bytes from it, and passes it up to layer 3.
10. Layer 3 (IPv4) on computer A sees that the destination of the IP packet is not computer A.
11. Computer A now proceeds to perform steps 1 through 7 again, except that computer A is looking for computer C. Computer A may broadcast the ARP request out of both of its interfaces.
12. Computer A now sends the packet to computer C.
13. Computer C, seeing that the packet is destined for itself, sends the packet further up the stack, eventually hitting the application layer, where a listening application may do something with the data.

That’s a lot, and it happens very quickly. Each of the computers will have an ARP cache, which stores the MAC address↔IP address mapping for a limited amount of time. When the entry goes stale, it will drop from the ARP cache, and the next time the computer has to talk to that IP address again it will issue another broadcast for the MAC.

Also remember that during all of this, layer 2 devices are invisible to the layer 3 protocol. Switches are smart enough not to interfere with the operation of things like ARP, by transmitting all broadcast packets out of all active ports (except for the port of origin); the same goes for all other types of MAC broadcast traffic. This is why DHCP works even when there is a switch between the computer requesting an address using DHCP and a system running a DHCP dæmon: the client initially sends the request to the special 255.255.255.255 IP address, and IPv4 stacks know that 255.255.255.255 is a network-wide broadcast address, so it in turn tells layer 2 to send the packet to the special FF:FF:FF:FF:FF:FF broadcast MAC address. Any bridge that sees that MAC address as the destination will send the packet across the bridge; remembering that switches are just a special application of a bridge, it does the same thing.

What next?

I am going to leave this as it is tonight. This concludes the first part of my post. You should feel comfortable with the concept of how the lower layers of networking work, and with how the Internet Protocol adds functionality to those lower layers of networking. If you do not feel comfortable with that yet (and you have read all of the links to Wikipedia and done research from there on out), then I want to know what your question(s) are.

I am not sure when I am going to write the next post in this series, could be tomorrow or in a week. In the next post, I will add more complexity to the network. There will likely be diagrams, because it won’t be simple enough to unambiguously describe in plain text anymore. After the next post you should be able to feel comfortable with diagrams, though, at all of the first three levels of the OSI model (and realize that they can—and very often are!—very different from each other).

Perhaps one key point to take from this post: Any time you have more than one logical network segment that is not connected by a bridge, you must have a router—and almost any computer operating system (certainly anything from the UNIX family) can be a router just fine.