Posted on August 5th, 2009 by matthew
By Declan Ingram
Over the past few years we have seen more and more automated scanning tools being used as the primary source of application assessment. A couple of years ago, when we were S-A.com, one of the guys did a very comprehensive test of all the available scanners, and the results were mediocre at best. In fact, as a result of these tests, we decided at the time that they added little to no benefit to our testing tool-chain.
Recently, with the enforcement of PCI Web Application Security Assesment requirements, clients need to have the coverage for all of their applications and do not have the funds available for full manual testing.
The three that we have been looking at recently are AppScan, Acunetix, and Burp Professional. Burp is a little bit different, in that it’s primarily a manual assessment tool with some scanning features.
We have been judging the quality of these products based on false positives, false negatives, and code coverage. The applications have all been web apps: HTML, JSP, ASP, PHP, old, new, good, bad, ugly, etc.
The results were……interesting:
- All scanners needed a lot of manual work to get any reasonable amount of code coverage.
- There were a huge amount of false positives.
- There were many false negatives. (Probably more than we know
)
However, these flaws can all generally (possibly excepting false negatives) be negated with a qualified person running the scans, and verifying the results. So this is really not a problem, right? I mean, it’s how the vendors advertise their low false-positive and false-negative rates.
The big problem, as I see it, is that these applications are not sold or targeted to specialist testers anywhere near as much as they are marketing to coders and auditors that do not have the skills to use them effectively. This negates the whole idea and provides a false sense of security!
The outstanding product here is burp, it’s a semi-automatic scanner, so it requires a skilled tester to use, but it’s a fraction of the cost and is targeted at the right market to get results.
Posted in Applications, Vulnerability Management, Web Application Security | 9 Comments »