Penetration testing is not a replacement for cybersecurity strategy

Paul McCarty — Lead Engineer at SecureStack

It’s our job to find good security practice and codify it. That’s what we do at SecureStack mostly: automate security into infrastructure. We do this a number of ways. For example, we build hardened operating systems that limit the attack surface of a server, and address a lot of the baked in security issues with default images you get from the cloud providers. We also use tools like WAF, dynamic firewalls, SELinux and others to do the actual securing part. We then audit it all and use centralised real-time logging and aggregation to make sure you can see all your security data. I think of it as a two part process with tools that do, and tools that quantify. So, generally there will be an initial audit, followed closely with a hardening and security implementation phase and then another audit round. That way we can see the difference between the first audit and the second and can gauge our success accordingly.

One of the things we do in the audit phase is run automatic vulnerability scans. SecureStack runs these scans, collects the data, indexes it and then alerts on any issues. You get a historical richness from these multiple automated scans which gives you insight into how your applications and environment are evolving. What we don’t do is run a one time vulnerability assessment, give you that report and call it good. We take your vulnerability scan data and add it with other source material so that you get that richness that is really required to make important security and infrastructure decisions. You don’t just find out why a port is open to the world. Instead, you understand what software opened that port, when it was opened and who installed the software.

So, why is pen testing not good security policy?

The problem happens when pen-testing is the only policy. Or used as the sole quantitive indicator of a security uplift project. I’ve seen this more frequently in the last two years: Companies will start first with a pen test and then never deliver anything else. They don’t have a well rounded strategy that includes real tools and real procedures. When the vulnerability report is delivered they address the critical issues by patching some boxes, or closing some security groups, and then by walking away from it. They don’t address the fundamental reasons they had the issues in the first place. You address those fundamental problems by using automation, good tools and most importantly by creating awareness and (hopefully) cultural change.

So what we are seeing is a deluge of pentests for companies of all sizes. In particular the smaller companies often treat it as their primary security focus. These companies are often the most cloud native and frequently are using containers and serverless. Securing these cutting edge platforms is difficult to be sure. But even when they do have significant compute running 24/7 it isn’t secured. At least the compute infrastructure isn’t. They might have CloudFront and AWS Shield turned on, but the vast majority of the time security is simply not a significant concern for them. But when they eventually land a big customer (government usually) they are required to prove that they are employing secure practices and producing a secure product. That’s usually when they turn to pentesting. The get a point in time pentest, remediate as best they can the findings and then forget about it until the next time someone requires quantification of their security practices. It’s a systemic problem exacerbated by the fact that in the absence of an IRAP or ISO27001 certification how does the customer discern their security? Pentests have become, unfortunately, the defacto standard of security quantification.

I see this particular theme in Australia more than I do in the States.

So, in closing, I am not saying don’t get pen tests done. I don’t think I need to say it, but I will anyway: penetration testing is an absolute necessity in todays world. What I AM saying though, is that if your companies strategy is to pay for pen tests and that’s it, no security tooling, security personnel or emphasis on bettering security, then your company is indeed in trouble.

Some warning signs…

1. If someone in your c suite has held up a pentest recently as evidence of “how secure you are” this article was definitely meant for you!
2. If you have budget for pen tests but not security products or personnel you are probably working for a startup that wrote its APIs in Powershell.
3. If you have no one on the payroll who’s job it is to secure your environment and aren’t employing a MSSP to do it for you, you have a problem. Your customers expect you to provide secure products and if you don’t have anyone on the payroll who is securing your products, you are cheating your customers.
4. If your company had a penetration test recently and you never got to see the report or if it took months before it was disseminated widely, your company is in trouble. There is no shame in advertising the security vulnerability report to your employees. That’s kinda the point. Bring people into a room and highlight what the report found. Talk about it. Otherwise, how do you plan on getting better?
5. If a vendor says they can address your remediation report with a single product, laugh and walk away. This is not possible. Wholistic security is a defence in depth strategy that is implemented in layers, which means you will be implementing more than one security product.
6. If a security vendor wants to sell you an appliance, or a software platform to ostensibly remediate security issues found in a pentest and they don’t include at least a months worth of customisation and configuration, you won’t be happy with the results. Security tools require integration into your existing infrastructure and toolset. This does not happen in a week no matter what they tell you.

That’s it for now. Hit me up on LinkedIn or Twitter at @eastsidemccarty if you have a question or want to give feedback.