The Problem with PEN Tests
This is a true story, the names have been changed to protect the guilty. Al Akazam (not his real name) is an IT consultant with a solid background in technology, and wants to expand his practice into network penetration (PEN) testing. So he downloaded a copy of Nessus, which is a powerful, open source, vulnerability scanner…and just like that Al Akazam was a PEN tester! Armed with this new tool, Al secured his first client, a financial institution. The institution was aware of the FFIEC guidance to periodically validate the effectiveness of their security controls through testing, and although Al didn’t possess audit credentials, nor vast experience with financial institutions, he seemed to know what he was talking about, and the institution engaged him.
Al got the institution to allow him to connect to the internal trusted network, where he activated his scanner and sat back to let it do its magic. An hour or 2 later the scan was complete, and Al had a couple hundred pages of results, some of which (according to his magic scanning tool) were very severe indeed. Confident that he had uncovered serious and immediate threats to the network, Al rushed the 200 page report to management, who were understandably very alarmed. Al completed the engagement secure in his belief that he had performed a valuable service…but in fact he had done just the opposite. He had done the institution a disservice. By not evaluating the threats in the context of the institutions’ entire security environment, Al misrepresented the actual severity of the threats, and unnecessarily alarmed management.
A vulnerability’s true threat impact, its exploitation factor, is best expressed in a formula:
Threat impact = (vulnerability * exploitation probability) – mitigating controls
Al simply took the list of potential vulnerabilities the scanner spit out, and without factoring in the exploitation probability, or factoring out the existing controls, changed the equation to:
Threat = vulnerability
What he should have done was take the threats he found, and evaluate them in the context of the institutions’ specific environment by ascertaining what preventive measures were in place, and how effective are they…i.e. the likelihood that the vulnerability would be exploited, and if preventive measures failed, what detective and corrective measures are in place to minimize the impact? The question Al should be addressing is not “what does my magic scanner say about the risk”, but “what is the actual risk”. Simply put, Al got lazy (more on that later).
What else did Al do wrong?:
- He didn’t start with an external scan. Since the external interface(s) are the ones getting the most attention from the hackers, they should also get more preventive, detective and corrective resources directed towards them. A risk-based approach demands that testing should always start at the outside, and work its way in.
- The institution gave him privileged access to the internal network, which is not realistic and does not simulate a real attack. Sure it’s possible that malware could allow an attacker access, and privilege elevation exploits can theoretically allow the attacker to gain privileged access, but is it likely? How many layers of controls would have to fail for that to happen?
- Again, he got lazy. He should have gone further in his testing by taking one of the most severe vulnerabilities, and tying to exploit it. Only then would management understand the true risk to the institution, and cost justify the allocation of resources to address it.
- He didn’t understand financial institutions. Bankers understand the concept of “layered security”, and how having multiple controls at various control points reduces the risk that any one failed control will result in an exploit. The vast majority of today’s financial institution networks are built using a layered security concept, and have been for some time. Shame on Al for not recognizing that.
- He presented management with a meaningless report. Instead of simply regurgitating the scanner severity ratings in the management report, he should have adjusted them for the control environment. In other words, if the scanner said a particular vulnerability was a 10 on a scale of 1 – 10, but the probability of exploit was 50%, and other overlapping and compensating controls are present, the actual threat might be closer to 3 or 4.
I’ve seen this scenario several times over the last few years, and in most (but not all) cases when the PEN tester is presented with the flaws in their methodology, they adjust accordingly. This is important, because a bad PEN test result has a ripple effect…you now have to expend resources to address issues that may not actually need addressing when placed in proper context. You have to present the report to management, with an explanation of why it’s really not as bad as it looks, and you have to make the report available to your examiner during your next safety and soundness examination. So for all these reasons, if you are a banker facing a similar situation, push back as hard as you can. And get outside help from an auditor or IT consultant to help make your case if necessary.
Are you a PEN tester or auditor? What is your approach to automated scanners and test results, do you adjust for the overall controls environment?
3 comments
Write a Comment
You must be logged in to post a comment.
April 5, 2013
Right on the money! You touched upon the consistent themes that I have reminded clients of over many years of reviewing their third-party scan results.
The scan you pay for only has value if the person who ran the scanning tool takes the time to interpret the results. I have encouraged more than one customer to get their money back when the only deliverable was a 900 page stack of scary scanning tool results dropped on their desk. A valuable report describes the realistic threat and where to find more information about understanding and remediating it.
Not everyone who can click the “scan” button is a security expert. Even fewer understand the risk management process for financial institutions.
April 5, 2013
Great post, Tom. Since you asked for feedback from pentesters on their approaches, I figured I would throw my two cents worth in.
First is the definition of a “penetration test” vs. a “vulnerability assessment” – get 10 quotes from 10 different vendors, and you will see wide variations in the use of these terms (and there is not necessarily one that is right or wrong). I personally see a penetration test as a blind attack with no credentials or other information provided by the network administrator. In these cases, a scanning tool such as Nessus will provide very limited results and the same results can generally be identified in a much stealthier manner with more targeted tools – think of surgery with a scalpel, not a machete. Vulnerability assessments, on the other hand, do generally involve credentialed scans and will identify network vulnerabilities that cannot readily be determined in a blind attack (other than potentially by simple trial and error). We generally perform hybrid “penetration test and vulnerability assessment” engagements, and the penetration testing phase is performed separately and prior to the commencement of hauling out the scanning tools.
Reading a credentialed Nessus scan report on a network of 200 hosts is a bit like drinking from a fire hose. While we do provide the scan results (along with other tool results) as appendices to our reports, I feel that a scan report should not serve as the only deliverable to a client when conducting these types of engagements. We take the scan results and combine related vulnerabilities so, rather than having 1000 high-risk vulnerabilities relating to missing Microsoft patches spread over 100 hosts, you have a single report item for “Missing Windows Patches” with a list of the hosts that need patching.
Nessus scan reports also have to be carefully vetted for false positives – an assessor is probably never going to catch them all, however they should attempt to a reasonable extent to identify and remove false positive items. if you see a report with a ton of “SMB Null Session” vulnerabilities included, you can pretty safely bet that it was not checked with a second tool to see that there could actually be information gleaned through a null session. I recently saw a report at one of my clients which had a high-risk Windows NetBIOS vulnerability reported on an external test (this is very unusual to see on the perimeter in modern networks so it peaked my interest). Looking further at the report, the IP address in question was actually a Cisco router!
I also agree that the risk of exploit should be considered in assessing the overall risks. We add a component for each vulnerability identified called “Business Risk” which is our subjective assessment on the impact on the organization based upon factors such as the types of hosts affected (are they printers or are they servers), the nature of the exploit (local privilege escalation vs. remote code execution), and sometimes mitigating controls. This risk level is averaged with the technical risk that is spit out by Nessus or other sources to arrive at a composite risk score. I must say that it is difficult for an assessor to completely consider mitigating controls in assessing risk, however it is a consideration when I am subjectively assigning these risk levels.
April 9, 2013
A very good write-up Tom. I completely agree and regularly get frustrated by the misconceptions between a true penetration test versus a vulnerability assessment and even a simple vulnerability scan (as performed by Al). Unfortunately, there are all too many vendors out there that offer this form of “penetration testing”.