The Problem with PEN Tests

This is a true story, the names have been changed to protect the guilty.  Al Akazam (not his real name) is an IT consultant with a solid background in technology, and wants to expand his practice into network penetration (PEN) testing.  So he downloaded a copy of Nessus, which is a powerful, open source, vulnerability scanner…and just like that Al Akazam was a PEN tester!  Armed with this new tool, Al secured his first client, a financial institution.  The institution was aware of the FFIEC guidance to periodically validate the effectiveness of their security controls through testing, and although Al didn’t possess audit credentials, nor vast experience with financial institutions, he seemed to know what he was talking about, and the institution engaged him.

Al got the institution to allow him to connect to the internal trusted network, where he activated his scanner and sat back to let it do its magic.  An hour or 2 later the scan was complete, and Al had a couple hundred pages of results, some of which (according to his magic scanning tool) were very severe indeed.  Confident that he had uncovered serious and immediate threats to the network, Al rushed the 200 page report to management, who were understandably very alarmed.  Al completed the engagement secure in his belief that he had performed a valuable service…but in fact he had done just the opposite.  He had done the institution a disservice.  By not evaluating the threats in the context of the institutions’ entire security environment, Al misrepresented the actual severity of the threats, and unnecessarily alarmed management.

A vulnerability’s true threat impact, its exploitation factor, is best expressed in a formula:

Threat impact = (vulnerability * exploitation probability) – mitigating controls

Al simply took the list of potential vulnerabilities the scanner spit out, and without factoring in the exploitation probability, or factoring out the existing controls, changed the equation to:

Threat = vulnerability

What he should have done was take the threats he found, and evaluate them in the context of the institutions’ specific environment by ascertaining what preventive measures were in place, and how effective are they…i.e. the likelihood that the vulnerability would be exploited, and if preventive measures failed, what detective and corrective measures are in place to minimize the impact?  The question Al should be addressing is not “what does my magic scanner say about the risk”, but “what is the actual risk”.  Simply put, Al got lazy (more on that later).

What else did Al do wrong?:

  • He didn’t start with an external scan.  Since the external interface(s) are the ones getting the most attention from the hackers, they should also get more preventive, detective and corrective resources directed towards them.  A risk-based approach demands that testing should always start at the outside, and work its way in.
  • The institution gave him privileged access to the internal network, which is not realistic and does not simulate a real attack.  Sure it’s possible that malware could allow an attacker access, and privilege elevation exploits can theoretically allow the attacker to gain privileged access, but is it likely?  How many layers of controls would have to fail for that to happen?
  • Again, he got lazy.  He should have gone further in his testing by taking one of the most severe vulnerabilities, and tying to exploit it.  Only then would management understand the true risk to the institution, and cost justify the allocation of resources to address it.
  • He didn’t understand financial institutions.  Bankers understand the concept of “layered security”, and how having multiple controls at various control points reduces the risk that any one failed control will result in an exploit.  The vast majority of today’s financial institution networks are built using a layered security concept, and have been for some time.  Shame on Al for not recognizing that.
  • He presented management with a meaningless report.  Instead of simply regurgitating the scanner severity ratings in the management report, he should have adjusted them for the control environment.  In other words, if the scanner said a particular vulnerability was a 10 on a scale of 1 – 10, but the probability of exploit was 50%, and other overlapping and compensating controls are present, the actual threat might be closer to 3 or 4.

I’ve seen this scenario several times over the last few years, and in most (but not all) cases when the PEN tester is presented with the flaws in their methodology, they adjust accordingly.  This is important, because a bad PEN test result has a ripple effect…you now have to expend resources to address issues that may not actually need addressing when placed in proper context.  You have to present the report to management, with an explanation of why it’s really not as bad as it looks, and you have to make the report available to your examiner during your next safety and soundness examination.  So for all these reasons, if you are a banker facing a similar situation, push back as hard as you can.  And get outside help from an auditor or IT consultant to help make your case if necessary.

Are you a PEN tester or auditor?  What is your approach to automated scanners and test results, do you adjust for the overall controls environment?

Print Friendly, PDF & Email

Join Our Community

Related Posts