This is the third post in a series on networking. If you are joining late, please read the first and second posts before going forward.

Today’s post is going to build on the previous two posts, and we are going to discuss the protection of your network. I’d like to start by saying that network protection takes place at multiple levels. In order to secure a network, you must work at all levels. It is absolutely impossible to secure a network by only looking at the network’s core infrastructure or by expecting that one can centrally secure everything. This is absolutely not the case. Security is relevant at all levels. In this post we will talk about security at many levels, including:

1. An individual device, such as a workstation, server, or networked appliance.
2. At the level of a single (physical) network segment, which includes all of the workstations, servers, and networked appliances on that segment. Recall that I discussed what physical and logical network segments are when I discussed bridges and switches in my first post in this series.
3. At the level of a whole subnetwork. For simple networks, only one subnetwork will be present, and so the security of the subnetwork will be the security of the network as a whole. For larger or more complex networks, this is security at the level of each individual subnetwork.
4. At the level of the whole network. This is effectively the same thing as security at the subnetwork level, although the business rules might be different (and thus, any firewalling or other security measures at this layer would also follow different rules). For example, subnets might be allowed to share privileged information or services amongst themselves, but such things might not be allowed to actually leave the network as a whole. Keep in mind that your network is just a subnetwork attached to whatever your upstream network is (which might or might not be the Internet itself, depending on your network structure).

We will also discuss things that are found at various layers of the network, and different types of systems: Client/server applications, Web applications (which are just a specific type of client/server application, when you think about it), authentication and authorization systems, file shares, printers, and so forth.

One At a Time: The Individual Device

Managing a network—especially one that has many network nodes—can be a very time-intensive task. There is much to do. Many of the tasks required to manage individual workstations can be automated (which becomes more and more useful as you have to manage larger numbers of network nodes). We have really only three types of network nodes on our networks:

1. Open systems, which we are able to work with in excruciating detail. These are devices that are running operating systems and software that we have the source code available for, including distributions of GNU/Linux, non-GNU Linux systems (such as those built around Busybox), FreeBSD, NetBSD, OpenBSD, and so forth. These are the systems that we truly have the power to manage literally every aspect of.
2. Semi-open systems, which we can do a great deal with: we can install software and maintain software on such systems, and we can follow the vendor’s requirements and security update procedures. Microsoft’s Windows operating system family and Apple’s OS X line of operating systems are examples of such systems. Also, operating systems that are built on BSD-licensed free software operating systems, but themselves are closed (proprietary) forks without source code available fall into this category. (Of note: Microsoft does make source code for Microsoft Windows available to people who have enough money to pay for the privilege. However, as I understand the terms of their source licensing program, one is not allowed to make modifications to the source code, and therefore is not truly able to manage the system with the same level of power or flexibility as any of the systems in the “open system” category.)
3. Black boxes, which are embedded network devices that we do not have the ability to work with at all. A limited configuration interface might be available. Such devices do not typically allow for the installation of new software, such as updated network stacks, unless the vendor decides to add support for such things. These are mostly network appliances, such as SIP phones and networked printers. There may be ways to work around some of the limitations in these devices, but this depends on the interfaces that the devices make available.

Preferences and idealism aside, realistically speaking almost all networks have systems that are types 1 and 3, and many networks have type 2 systems as well. Most small office networks consists of mostly type 2 and type 3 systems, which type 1 systems usually running things “behind the scenes”, if they have type 1 systems at all. We will focus on types 1 and 2 in this section; most type 3 devices do not have security parameters to speak of and must be protected by the network itself. There are, of course, exceptions; we will not concern ourselves with these. We will also make the assumption that these type 1 and type 2 devices are computer systems of one sort or another: desktops, laptops, servers, netbooks, and some types of smartphones (such as those that run Android, which is a non-GNU distribution built around the Linux kernel).

In order to secure a network, the first step is to ensure that all of the devices that are on the network are themselves secure. This implies that there must be some means by which devices can be rejected from the network if they are not, but we will cover that when we take a look at security at the subnetwork level. Individual device security includes the following items (but is by no means limited to only these items):

1. The security of the operating system kernel itself. The operating system kernel provides critical system services; it is usually tasked with the management of hardware resources. It provides a “gateway” to all of the hardware devices in the computer itself, including audio, video, storage devices, RAM, I/O ports, network interfaces, communications buses, hardware cryptographic engines, and so forth. Most operating system kernels also provide the primitives required to handle networking, beyond talking to the network interface. The Linux, FreeBSD, OpenBSD, and NetBSD kernels all handle IPv4 and IPv6 as part of the operating system kernel itself, as well as things like virtual network adapters (which make tunneling possible) and IPsec. The operating system kernel does an awful lot for us! If the operating system kernel has a security vulnerability (particularly one that permits privilege elevation without proper credentials or that permits a user to crash the kernel, resulting in a denial of service) then it becomes possible for anyone to bring the whole system down, taking all of its software along with it.
2. The security of the core system libraries and software; that is, the userland software that provides the other half of the operating system itself. For most systems that use the Linux kernel, this part of the operating system is provided by the GNU project: the C library, all of the core utilities, the basic shell, and several other things. There are also several userland utilities that are maintained by kernel developers that provide a means by which the kernel can be managed and maintained. All of these core items are part of the operating system (and the interface that it exposes to application software). Application software doesn’t worry about performing tasks that are handled by the operating system, such as managing network interfaces (though application software might be concerned with monitoring such things). What this means is that if there are security problems in a function in the system’s C library, any application software (or indeed, any software at all, including core operating system management utilities) that use that function are potentially vulnerable. This makes it important to keep an eye out for reported vulnerabilities in such things.
3. Finally, there is the application software running on the server itself. If the software running on the server is not programmed with security in mind or has not been throughly audited (and even if it has!) it is entirely possible that there are (or will be) security problems discovered. If you are running software that is created yourself, then you need to be especially diligent and constantly be reviewing your applications for security problems. If you are running software provided by someone else, you should keep up on security vulnerability reports, such as those from CERT’s vulnerabilities database. In fact, if you manage any type of network at all, you should be subscribed to the feed (and regularly keep up on it)  so that you’re informed about new vulnerabilities as information on them becomes generally available.

There are also other mailing lists and feeds around the Internet that are focused on reporting security vulnerability information; I would recommend that you be subscribed to as many of them as you can be, and somehow filter them so that you see only the vulnerabilities that are relevant to the systems that you run.

Hire (or Create!) Help If Needed

If you run a network that has more than one or two dozen nodes, it starts to become difficult to manage it all by yourself; not necessarily because of the number of devices involved, but because of the number of activities that are involved in keeping them secure. Certainly if you manage everything manually it becomes extremely difficult. You have system logs to go through, problems to track down and fix, intermittent issues sometimes from seemingly nowhere. For any size network here are some good things you can do in order to be aware of the “security health” of the systems on your network:

• Monitoring software, such that you can monitor uptime, memory consumption, disk space usage, inode usage (when relevant) and other aspects of all of your individual workstations. Then set the monitoring server to call your attention to any situation that arises that requires human intervention. For example, you need to know if the filesystem on any given system is more than a certain percentage full (how much varies on the system and how quickly the data growth is on that system; some systems might be fine alerting you at 95% full, others you might need to know significantly earlier, say when it is 75% full). You can monitor many different aspects of a computer system, and these can all be early warning signs.
• Aggregate your system logs and event logs in a single place. Have a monitoring system available that can issue alerts when extraordinary events take place. Those extraordinary events might be early indications of a security issue. Learn what’s “normal” for your particular network (every network is different!) such that you can filter out normal system activity. But do not throw away log entries. Archive all of them, somehow or another, since they may be important for forensic analysis later. Aggregation of log files also makes it much more difficult for an attacker to cover their tracks by, for example, overwriting incriminating log entries on a system; perhaps a system’s individual log files are tampered with, but the aggregated copy should be in a safe location and not use the same authentication credentials as other systems on the network, in order to raise the difficulty even further.
• Have regularly scheduled “downtime windows” that can be used for things like applying patches. Humans aren’t perfect, and are sometimes even malicious; thus it is necessary from time to time to apply patches to the software that you run. If all the software you are using comes from a distribution (such as Ubuntu) then this process is made significantly easier, because you can perform all patches and issue a reboot command in a matter of minutes.
• If you have many machines, keep a local mirror of the package archives for the operating system distributions that you have deployed. If you use both Debian and Ubuntu on your network, for example, and you have more than half a dozen or so machines, then it’s a good idea to have a local copy of the package archives for both. You will need a decent amount of space (between 30 and 70 GB for each supported release), but it will significantly improve the time it takes to roll out updates and deploy new systems, as well as save bandwidth (since you will only have to download the updates once, instead of n times, where n is the number of systems you have deployed).

If you get to the point where you are spending all of your time performing maintenance of one sort or another, then it is time to either hire more help—or automate more maintenance. But be careful! There is only so much that can be automatically done without the supervision of a human being to notice that something just isn’t quite right…

Create Update Alerts for “Helpless” Devices

Nearly every network has at least one (and frequently, many) devices that are “helpless”. That is, they are not able to check for updates for themselves. That means that it is up to you in order to check for updates on its behalf. You can do this one of a few different ways. I am a huge fan of saving myself time and automating things where possible, of course. However, that is not always possible; your automation might have to be as simple as “Create an issue in our issue management system every x days to remind me to check for updates for device d“, such that at least then you have the ability to be automatically reminded. You can then check and close the ticket when you find that no update is necessary, or close the ticket when you find an update and create a new ticket that has the goal of actually applying the update.

Audit Updates

When you apply updates, if you have the ability to do so, audit them. See what vulnerabilities they fix. See if the vulnerabilities that are fixed are all of the known vulnerabilities. If you have the ability to do so, review the source code differences so that you can see what the changes are. You never know, you could find a security bug (or other bug) introduced in the patches.

Automate Configuration When Possible

Network security problems at the device level do not just come from bugs in software. They can also come from configuration problems. Perhaps a system is—or a family of systems are—too permissive and do not correctly apply business rules. Perhaps the business rules themselves are flawed such that the translation from business rule to configuration is faithful, but creates a security problem. In any case, do not manage system configuration manually if you can possibly avoid it! If you have a medium to large sized network, odds are that you’re going to have multiple systems that have similar configurations.

Use a centralized system in order to deploy configurations to systems in your network. Also, this way, you only have to back up a single configuration database (this simplifies your needs for recovery later should you become the victim of an attack or some other form of disaster). It also gives you the ability to quickly get back on your feet when a system unexpectedly dies—and the ability to audit change control is extremely useful, as well.

Disable and/or Remove Unnecessary Dæmons/Services

Some operating systems come with a number of dæmons (POSIX systems, including the UNIX family) or services, background processes, or whatever. These things can, for example, perform automatic system maintenance. Or they can open up ports and accept commands. Whatever they do, if you do not use the services in your own network, disable them. Anything that you disable, make a note of, and have your monitoring system always validate that it remains disabled. An alarm should go off if it is ever re-enabled (unless you have to re-enable it for some reason, then you should of course tell your monitoring system to not alarm for it). This reduces the number of vectors available to mount an attack. If you do need such a service, but it does not need to be exposed to the world, use a firewall to keep it in (we will talk more about that when we get to subnet and network level security). If you don’t know what a service is or does, research it—ignorance is absolutely intolerable when it comes to working in security. If you don’t already do research on a regular basis and you are charged with the management of a network, that is something that you should seriously consider changing. Seriously.

Network Segments

Security at the level of a network segment is often synonymous with security at the subnetwork level. Of course, as we have already read, multiple physical network segments can be joined together in two different ways (well, more, actually, but we’ll ignore this for now) such that the multiple physical network segments become a large logical network segment.

At this layer, the most important thing is to know your network well. Understand the physical and logical topology of your network. If you find that there are groups of systems that do different things and they are all sharing one logical network segment, take the time to consider whether or not you should replace a bridge interconnection with a router.

To a certain extent, the security of a network is the performance of the network. Denial of service can result in severe business losses, for example. While relatively uncommon, it only takes a single misbehaving (or penetrated) system in order to bring down a network segment. Consider using routing in place of bridging unless there is some significant advantage to bridging between segments.

Subnetworks and the Overall Network

And here we are.  Security at the layer of a single subnetwork. If you have only one network with zero subnetworks, then you only have one subnetwork: the one that is given to you by your upstream network. Every network (except for the root network) is a subnetwork. Every subnetwork is a network. That’s all there is to it—so really, subnetworks and networks are treated the same way when it comes to evaluating their security.

A network has one or more edges; there can be inner edges and outer edges. Outer edges are gateway points to parent networks; a router lives there and it is usually a default gateway. Inner edges are the same, except they point to a subnetwork that is logically “beneath” the current network. For packets to get to the inner subnetwork from the parent network they must be routed through the current network (in other words, the current network provides transit between the two networks). It is also possible for a network to provide transit between two (or more) outer edges. And just to blow your mind a little further: it doesn’t really matter much to the computer what we call an inner edge or an outer edge. It’s all the same to it; it’s just packets moving around from source to destination via hops along the way. The abstraction of a “tree” structure to the network, or the abstraction of “inner” vs. “outer”, is merely there to make it convenient for us simple-minded humans to talk about and really has nothing to do with how the system actually processes and routes packets.

In order to secure a (sub)network, we must be sure that:

• The only connections allowed from outside the network (e.g., originating from the other side of that network’s edge router or routers and terminating on the inside of that network’s edge router or routers) are those that are allowed by business rules.
• The only connections allowed from inside the network to the outside (e.g., originating from the inside of the network’s edge router or routers and terminating on the other side of that network’s edge router or routers) are those that are allowed by business rules.
• Services that are internal to the network are not exposed to the “outside world” (that is, not reachable from outside of the network’s edge router or routers).

When you have a network that has multiple subnetworks, it can be the case that services on one subnetwork should be accessible to a “sibling” subnetwork in the same administrative domain (and sharing the same parent network as implied by the term “sibling”). In that case, some of the responsibilities for the rules listed above are delegated to the parent network’s edge router(s). Of course, it can get to be far more complex than that, even. Sibling subnetworks could have rules that only permit a pair of siblings to share access to particularly exposed services such that while the sibling networks can access the private services on each other’s networks, connections are disallowed from other sibling networks or even the parent network.

Another way to say it: the design of a network structure can be as simple or as complex as required to support business rules. As long as the logical structure of the network (which, as you will recall, need not precisely match the physical structure of the network) is compatible with the rules that are to be enforced, everything is nice and easy. Where things get difficult is when the design of the network is insufficient to satisfy all business rule requirements. One example of how this can happen is normal business growth; e.g., when new business rules are introduced that are incompatible with the present design of the network. And in that case, the part of the requirement for adopting the new business rule(s) includes a redesign of the network’s logical structure—which might be simplified by using tunnels, for example, to create another subnetwork altogether.

Summary

The tools that we have in our virtual toolbox for network management are many—and they vary widely. In our arsenal, we have:

• The wires themselves (and their physical layout)
• Workstations
• Servers
• Physical network interfaces
• Bridges
• Switches
• Routers
• Firewalls
• Network prefixes
• Logical (virtual) network interfaces
• Tunnels
• Encryption
• Subnetting
• ALGs (Application Layer Gateways) and proxy servers

… and more.

Alone, each one of these tools does nothing significant. Together, they do absolutely amazing things—and can do so in ways that are infinitely flexible (and yet surprisingly simple).

What’s Next?

I plan on writing at least another couple of posts in this series. I believe that I am mostly done with the theory component, and so the next post or two will deal with actual networks. In the next post, I’m going to discuss the layout of the network I spend the most time on: my own. That’s the only thing that I know for sure at the moment. From there, it’s pretty much anyone guess.

If I have left you behind, or if you are confused, please feel free to comment and ask questions on any of the three posts in this series. I will get back as soon as I can after seeing the question.

Until next time.