It is not uncommon for my team to get calls from our sales teams in competitive evaluations asking us questions like - “Does the IPS protect against CVE-XXXX? How can we verify it?" Recently, TippingPoint was engaged in one such evaluation at a customer location and the subject of independent test lab reports was brought up. Typically, customers use a number of data sources before picking which vendors they will evaluate. It helps them eliminate chaff and spend their valuable time more effectively. Beyond test reports they use information from industry analyst reports, vendor web sites, and vendor customer references to get a balanced view of the vendor’s product offerings. But let’s discuss test reports here and their role in the decision making process in a little more detail...

Two types of testing approaches have emerged over the years. In one case a vendor will fund tests that are iterative until the vendor is “approved” or “passes”. Another case is the report is still funded (by whom sometimes we don’t know) but the vendor undergoing the test does not participate in the testing at all and does not have visibility into the exploits or attacks being generated to the device under test (DUT). Additionally, the vendor is not given the opportunity to respond to the results until after the report is published publically. The danger here is that customers can sometimes view these two testing approaches as the same, mistakenly considering the results apples to apples (which they are not). Obviously, in one scenario the vendor is able to make adjustments to the configuration until a “pass” was achieved. In the other, no opportunity was given and the results are what they are...

In either case, it’s important to look at where labs tests ultimately differ from real-world customer scenarios and protection.

Let us say vendor X's "shoppingcart.asp" contains a SQL Injection vulnerability that is triggered when the first character to its "item_id" parameter is a quote(').

For instance, the URL #1 
http://ignorant.vendor.com/shoppingcart.asp?item_id='
will trigger the SQL Injection vulnerability, and so will the following URL#2
http://ignorant.vendor.com/shoppingcart.asp?item_id=’;SELECT * from users...

Now, let us look at what choices we have on the IPS for filtering and protecting that ignorant vendor from the vulnerability:
(a) IPS detects shoppingcart.asp that is passed an item_id parameter with the first character as a quote
(b) IPS detects shoppingcart.asp is passed an item_id parameter with the first character as a quote or and the length of the item_id parameter is longer than 8 bytes

Now, one may wonder, why would one prefer option (b) for filtering? There, my friends, is the gospel for IPS filters: No false positives!!! I don't know how many Tom, Dick and Harry vendors around the globe (I have to worry about the globe because with the exception of Arctic and Antarctic, our IPS products are deployed in large numbers in all other continents) create shopping cart products with shoppingcart.asp as a main script and item_id as a parameter to it. URL#1 is definitely going to trigger the vulnerability but at most what may happen is that the web server will display an error message from the database. That surely does not give the attacker the satisfaction they are seeking! The attacker will not be able to resist the temptation to tamper with the backend database and the commands he/she will enter to tamper with the backend will usually be longer than 8  bytes.

As you can see, sometimes there is little difference between malicious and legitimate traffic.  Unless you take time to really understand the traffic and create the applicable filter you’re likely to stop the “good” traffic, and that can be bad for business.  In the case of online commerce, this could mean millions of dollars in lost revenue in a matter of minutes.

Would TippingPoint enable such a filter in the "Recommended" settings? I am sure there would be differences of opinion in the "Weekly Filter Review" process over this issue amongst security researchers. But, most likely output would be "No". We will advise the customers that if they are running a shopping cart from Vendor X they should enable the filter.

Why is all the above relevant?
A testing house may test the IPS with URL#1 and come to the conclusion that the IPS fails the test. If the testing house at least asks the vendor why they are missing the attack - the vendor can at least defend their stance for creating the filter in a specific manner and offer an explanation (which is what any reputable vendor would do for its customers by the way.) 

So, it’s important to put lab testing into context and fit their test results into a comprehensive vendor selection strategy. We truly believe that each customer environment is different in many ways such as topology/applications, mitigation goals, and exposure to potential threats. These differences need to be considered when evaluating solutions that mitigate threats. Customers should leverage information available to determine which vendors could potentially help solve the problem. But it’s testing the vendor’s solution in the unique customer environment that will provide the most relevant data as a basis for product selection.  We at TippingPoint take security research very seriously, and we continue to invest in this area as a foundation of our product offering. For example, did you know that the ZDI program has grown from ~600 researchers in 2008 to well over 1000 in 2009? We’re also increasing our DVLabs team this year and actively hiring now! The commitment we continue to make to our security research means our customers reap the benefits of protecting their critical assets/applications and mitigating overall risk to their business.