Continuous vulnerability and configuration scanning is a boon to any organization trying to keep abreast of the security of their network assets. It's like having a penetration test done on a constant basis. But it also serves as a way to aggregate all the vendor and third-party advisories that impact your assets. If you have an automated scanning tool like Tenable's Nessus feeding into a ticketing system, you can deliver the notifications directly to the asset owners. Taking the CIS hardening benchmarks that work with this sort of tool, you can be sure of keeping up to date not only on vulnerabilities but also optimal security configuration settings available for your assets. "Hardening" systems in this way ensures a much better stance against unauthorized access to our IT resources for the purposes of stealing or altering data, obtaining intellectual property, misusing resources for unintended purposes, causing outages, monitoring personnel, etc. It's a high-value activity and worth investigation by any organization with a dependence on reliable and high-integrity IT systems.
On paper, it all sounds simple. But realities are always different, and I want to explain the realities of an automated vulnerability scanning and ticketing system to highlight how it's never really so simple when it comes to IT. This is written from the perspective of someone who has implemented such a system in response to an innocuous audit finding, and who witnessed it get out of hand. Career-altering out of hand. Also, I want to preface this by saying that I've been a fan of hardening systems for more than 25 years and in fact posted an early guide to hardening UNIX systems in 1998. It is my intention to encourage hardening wherever possible and to attempt automated scanning.
The acronym VUCA has been around for a decade or so: "volatile, uncertain, complex, and ambiguous". I find it useful from time to time to explain IT activities in light of these four adjectives because they're nearly always present and when you start an automated vulnerability and configuration scanning system you can practically hear the hoofbeats.
Again, our objective is to harden our systems to published standards for the purpose of limiting unintended access to, alternation of, and mis-use of our IT systems. Hardening an environment of typical size for a moderately large company (say, 1000 employees) is challenging enough just because of the size of the organization and the difficulty of automating anything. More on these in a moment. But consider these factors:
In sum; we're trying to automate against a moving target and there are several axis of motion.
But now let's consider that what we're doing can disable business functions of the technologies in play that are not well understood by the technologists that manage them. Turning something off because we don't know that it's used is bringing about exactly the sort of denial of service that might be the intent of malicious party trying to exploit a certain configuration.
And finally: you're not done when you've hardened your last Windows system just before the deadline at the end of Q1. Because you're introducing several new Windows systems next quarter, and a combination of vendor and internal requirements means you're adding some from that version you excluded from your hardening!
To revise my summary statement: we're trying to hit a moving target (imagine a carnival pellet-gun game, where the targets move in unguessable directions), and the function of that target is critical to our business. We have to be both precise and sensitive. And we have a pellet gun.
When it comes to hardening, there really is no 'definition of done'. Unlike with vulnerabilities, we don't get a "severity rating" with the controls in a hardening standard because with configuration settings the options are all there on purpose; in certain applications all of the options have a purpose. They have to, or they wouldn't be there. There is also no baked-in assessment of the relative priority of the configuration issues listed in the guidance; in fact the benchmarks break their guidance items into two categories: those that are not expected to impact regular use of the system, and those that are expected to impact regular function.
And because we don't know what configuration items might some day lead to a compromised system, we might be tempted to do everything in the benchmarks. But mindlessly pursuing the massive scope of systems hardening in a head-long, full-scope way might not really solve anything. We just don't know which of the many hardening controls in the standards is the magic bullet because that's unknowable given the particulars of any unique organization.
In sum; in addition to trying to shoot a moving carnival game target in a way that won't hurt our target, and no one knows when we've actually scored the hit until someone in the future determines how to prove a negative (you haven't been "hacked"). Wait, did I hear the whinny of horses outside the fair ground?
A corporate LAN is made of more than just Windows servers, so you'll want to look at Linux and perhaps "mainframe" systems, network devices, databases, middle-ware, web servers, and anything else that's present. And I mean anything: "integrated lights-out"; terminal servers, backup devices, storage arrays, VM hypervisors, anything. There might easily be 25 different classes of device when you count multiple versions of an asset (such as Windows 2008, 2012...). While some network device templates have as few as 30 controls, operating systems have 500.
Naturally, once you automatically scan for vulnerabilities and configuration matters, such an automated ticketing system could potentially generate a lot of tickets. Even setting aside vulnerabilities, it could easily be the product of the number of devices x the number of controls in the template you're using for the automation tool. If for instance you have 500 configuration controls in a Windows template, and 300 Windows servers, you could have up to 150,000 control items to scan. Not something you want being fed into your ticketing system and assigned to a handful of systems engineers! Or worse, non-technical "product owners".
Within and across different device types, there is a lot of variability; it's by no means a one-and-done. Countless configuration items are fairly unique due to the situation of the device in question: a print server on the cloud is different from a file server on-prem, and both are different from an application server or a global email server... Your environment will use certain network protocols and server features and not others, and the choices in your firm won't match those of all other firms: so no one benchmark file for a very specific system will suit all corporate environments. (Here it's important to understand that the standards are written to granularity of "Azure-based Red Hat Enterprise Linux v9 DNS server".)
In sum, once more; we are trying to shoot all of several moving carnival targets in a way that won't hurt the target, as those targets swing and whoop their way through the entire carnival grounds. And we may never know that we've actually scored the hit.
In an review exercise with a CTO, I showed how the CIS benchmark for nginx—a popular web server software product—insisted that only the pre-compiled version of the software that comes with the operating system's distribution should be used. At least it said so for the first few controls. But then other controls requires that the web server be compiled with certain options activated or deactivated. So; which is it, pre-compiled or locally compiled? It can't be both. Were these contradictions reviewed with the authors of the benchmark? Are we meant to install a compiler on each host to follow the second path? Because that's contrary to the control files for Linux.
Obviously it's a challenge automate something that needs human judgment calls. I'll also add that the fact that a division manager and the CTO (a VP-level executive) had to have several investigative meetings on such a subject should warn you how easy it is for these things to get "out of hand", as I warned above.
I have seen security professionals insist on answers for all of the controls in a file, even if it means explaining why obsolete control issues are not enforced. At a certain point—and there are countless potential pit-falls like this—the exercise is no longer only of uncertain value; it's not even clear what the objective even is. Is it hardened systems? Is it a smoothly-functioning ticket-production system? Is it a library of documented exceptions?
So, in final summary; we are trying to shoot all of several moving carnival targets in a way that won't hurt the target, as those targets swing and whoop their way through the entire carnival grounds. And we may never know that we've actually scored the hit and now we're not sure the judges themselves are sure. Here's your pellet gun; the game started when you brought your first system online and your current score is zero. Oh, and you're doing this on horseback because those four friends of yours insisted.
I've tried to show that an enterprise-wide systems hardening effort is volatile, uncertain, complex, and ambiguous. It's also costly in terms of man-hours and in lost opportunity to address other matters. I have literally lost sleep over this sort of program—and not just a little. Before you put someone's eye out with that pellet gun, please consider the following.
Understand your objectives. Vulnerabilities should be addressed wherever they are found, prioritized by their severity. But hardening is a lengthy process, and I recommend that an organization should only put in the hardening effort on those systems that have a future. Aside from the hardening time, there will be substantial process overhead: the asset owners, security staff, and the auditors will all have opinions on the controls in the templates. The substantial overhead of an agreed hardening approach on each class of device, plus the maintenance of the automated scanning, means that significant time can go into any given system. This should not be spent on hardening items with indeterminate life-span; if you're going to retire it, focus on those modernization activities instead.
Start small and go slow. If you attempt to do too much, you can run into:
Whatever happens, do not risk damaging the harmony of your organization over this. Do not let this turn into a politically-charged affair. Do not let this become an ordeal. Do not let it become resented and cause people to conclude that security is a pointless hassle with no value. Because they will: I have personally witnessed a CISSP swear off any further security work because of this automated hardening challenge.
I've already mentioned this matter but I think it needs elaboration: do not confuse configuration and hardening items with vulnerabilities. I will attempt here to explain the fundamental differences:
Intentionality: vulnerabilities are unintentional matters that are unknown to the manufacturer when the technology is released. Hardening configuration items are understood by the manufacturer and the industries that use them—they are designed elements included in the product design to include the wide array of uses that the technology is put to. This design flexibility is a fundamental design element of "open systems" such as Windows, Linux, and the mixed networking environment we employ.
Business relevance: the hardening standards divide their constituent items not by any kind of "criticality" but rather by potential for restriction on a typical business use. Vulnerabilities do have a criticality because they can only be abused; they do not have business use. One doesn't usually fail to patch a vulnerability because of a business use of the vulnerability.
Proportionality: Hardening items typically have to be used in conjunction to allow an exploit. This is because severe hardening matters are understood by the manufacturer and are mitigated by other components of the manufacturer's product design. Vulnerabilities, by counterexample introduce unknown and potentially unlimited negative impact. In our experience, the confluence of several configuration items were decried as "equal to a vulnerability" in severity and therefor urgency. Nowhere could we find this being supported in the literature.
Time sensitivity: Due to their use as attack venues with currency to their criminal and military users, vulnerabilities are most valuable when unpublished. These so called 0-day exploits are highly sought-after and actually have a market value. By contrast, configuration items can endure for decades because they do not have the currency of vulnerabilities.
Scope: Configuration matters abound, vulnerabilities should be orders of magnitude fewer in number. I've seen an organization turn automated configuration scanning loose and generate more than four million tickets. Scope is factor because of the seeming possibility of dealing with some thousands of vulnerabilities and the clear impossibility of dealing with millions of the same thing.
It is vital that you record, track, and communicate vulnerability matters separately from hardening matters. The same tool (e.g. Tenable's Nessus) may be feeding similar-seeming matters into the same ticketing system, but the entire organization has to agree that these are not the same beast and cannot be treated the same way.
Kill false positives and other scanning errors before they discredit the entire undertaking. Due the always-on nature of automated scanning, false positives will be perpetually reported. Incomplete scanning results when the Tenable software exits upon failure. The many manual-only matters in the benchmarks with reduce the sense of scope completeness.
These problems will engender confusion and disagreement as it obliterates clarity. These problems can readily make progress impossible to measure. These matters destroy the motivation of the subject matter experts.
In the "go slow" recommendation above, I suggest that the process of automated scanning should be kept to only those configuration items that can be reliably scanned. But another process should exist to decide on how to deal with things that will continually appear as false positives. Do not start automated scanning until the quality of the outputs is there.
Instead of Tenable, use Ansible or another automated configuration tool to enforce the configuration of your assets remotely. You can feed Ansible the same rules—those CIS standards. But Ansible lets you not merely report on the deviations but also to automatically eliminate them. Ansible allows you to draw up lists of target systems, so you can solve for the problems of unique servers or devices by segregating those disparate systems those groups, and then having a unique configuration set up for that group.
We've found that by putting our configuration and system lists into version control and then making changes by committing them to the repository, we can show when a change was made, or when a system was added to a certain configuration classification.
Starting with enforcement rather than detection is an audit-ready and deterministic way of doing things that will save you a lot of grief. We started using Ansible in this fashion with our network devices, and naturally added our Linux systems. For Windows there are probably better tools but we are looking at those as well and I profess that I'm not well placed to speak to the technical dealings of Microsoft's world. Even the various virtual and physical hosts that have been running this website for the past four or five years have been under the management of Ansible; I would not even consider running something as simple as this toy without it.
I've mixed my metaphors pretty mightily in this article, in part to try to inject some levity into something that literally landed one of my staff in front of a company-appointed doctor. It's very easy to get this wrong, for very clear reasons. And getting it wrong can come at a high price. So I urge everyone do this with great care and attempt this vital security activity in a way that will generate results and not dysfunction.
I'll also offer my time to anyone who needs to understand more. Please reach out if you feel you need more understanding before you pick up that pellet gun, mount your horse, and wonder how it ever came to this.
*Fifth recommendation added 2023.03.26—not sure how I missed it!