NSS Labs IPS Test: Why We Should Not Shoot The Messenger
Yesterday something big occurred in the world of information security. Something which is bound to have a massive impact on the world of Intrusion Prevention Systems (IPS). Perhaps IPS units and vendors will never be viewed in the same light. Perhaps that’s a good thing. I suspect the use and application, of these devices in addition to the ever coveted vendor-client relationship, will require some adjustment. It would were in the market for IPS appliances or worse yet, already owned a given number of them and they failed to perform accordingly in testing. NSS Labs released a new IPS test report which is raising a great deal of controversy with respect to some industry players while seeing others reap the rewards of delivering product that performs as promised. For the ones which did not perform as expected, I believe the words of the immortal Desi Arnez character Ricky Ricardo are appropriate “..you got alot of explaining to do”.
Utilizing a revised testing methodology designed to ensure the highest degree of integrity possible, NSS Labs employed 1,159 live exploits against the vendors who submitted to the tests. The tests rigorous and beyond simply important. They are necessary for ensuring the the rubber meets the road with respect to claims being made by vendors with respect to their products effectiveness. Without tests of this nature, individual organizations would be left to either employ internal testing methods (which by the way, I personally advocate in addition to third party tests), or rely solely on the word of the vendor which, given the nature of the results of the current NSS report, may not be enough.
We as an industry have an obligation to ensure that what is being developed and designed meets the expectations of the target audience in addition to those set forth by the vendors themselves. I found it interesting to see the response to the tests in speaking with peers throughout the industry. All expressed concern about the potential implications for environments where those vendors which did not fair too well are employed. Some felt the results should be released in a bit of a vacuum, others felt the delivery was spot on. Whatever your feelings on this topic happen to be, my assertion to you is this: providing appropriate disclosure protocol was followed we should not shoot the messenger.
Who should be shot? Tough call. I believe that this can be answered on a case by case basis however there is an expectation that all vendors currently marketing and selling solutions today meet at least a minimum criteria deemed acceptable to themselves and third party testers such as NSS Labs. In perfect world this would translate to 100% effectiveness and genuinely easy process. As this is not a perfect world we can only assume that the vendors who are designing these solutions are doing so in order to address evolving threats identified within the threat landscape in the most comprehensive manner possible.
NSS Labs has always been known for integrity in their testing and I believe this test (in addition to the work they’ve done in ensuring maturity and growth in their processes over the course of the last two years demonstrates this), demonstrates this in spades. Demanding that integrity be seen within products being sold for the express purpose of defending against advanced malicious code & content and next generation threats is not only intelligent it is the expectation.
I believe NSS Labs should be applauded for their efforts in pursuing integrity driven testing methodologies and results. More of this is needed within our industry to ensure the greatest degree of care is taken in selecting a vendor product when the time comes to do so. I do not believe that the NSS Lab team should be critiqued for this. Was there a better way of approaching the disclosure of the results? I am not sure. I personally believe bad news, unlike wine or cheese, does not get better with age and as a result (provided proper disclosure measures were taken with the vendors), there is a responsibility on behalf the NSS Labs team to report the truth. This ensures their independence and credibility. To do otherwise would be a disservice to themselves, the consumers, the vendors, and the industry as a whole.
Comments
Leave a Reply
Jerry on 12.08.2009
Well said.
Scott Lupfer on 12.08.2009
Will,
Well stated, let me follow up a bit on this and maybe we make our collaborative effort a “part 2″ to this.
I haven’t read the report myself and don’t know who tested to what level so I have a degree of ignorance going into my comments. However, if the test was truly independent and NSS Labs was not commissioned (READ: Paid) by any of the vendors to conduct this test, then none of the vendors should be upset with how the results were released or documented. There aren’t any disclosure rules around product performance, nor should there be when we are talking about testing a product against what a specific vendor is advertising.
However, there may be concern on the part of the customer who has implemented these technologies if they indeed are not protecting the customer as well as has been claimed by the vendor. If these weaknesses get out, then the theory might be that an adversary could use the attacks that the system is most weak at protecting against the network. That said, there are a lot of “ifs” that must be qualified within that statement before it is valid and “If ifs were fifths we’d all be drunk” so to speak.
There are 3 specific areas that MUST be tested or examined as part of a security product review:
1) the product’s “out of the box” security content must be tested against current, real threats. By out of the box, I mean that the vendor shouldn’t be allowed to submit a “test configuration”, it MUST be what every customer would receive that same day.
2) the vendor must be tested for response to deficiencies, meaning that if the particular product is found to be deficient in preventing or identifying a specific attack set or threat, the vendor must be tested for a response time. This effectively tests the vendor’s ability to respond, in near real time, and provide their customers a content update.
3) the product must be tested for vulnerabilities. Can the product itself be attacked and/or compromised.
Parts 1 and 2 should be subject to immediate public disclosure through any reporting function, whereas part 3 should go through responsible disclosure channels that the industry has in place today.
Again, I’m posting this without knowing any of the results, but am glad that it appears all of the major vendors were tested. Let’s see the spin start.
Scott
Will Gragido on 12.08.2009
Thanks Jerry. I’ve been thinking quite a bit about this since the news broke on the report. Though I think many folks are torn with respect to the matter, I believe we need to have voices such as this (whether its NSS, US Cert, etc.). Provided protocol is followed and a vendor is given ample time to digest the data provided to them, (remember these are not vulnerabilities NSS is talking about this is the failure to block or mitigate exploit based attacks which are supposed to be blocked based on the vendors specs and research), I see no issue with it personally. It will be very interesting to see the “lay of the land” going forward after this among the vendors.
Will quality improve? My prediction is that yes, quality will improve if vendors hope to maintain their footprints and customer base. Will this likely occur again? Perhaps, but whether or not it occurs again in the same way isn’t the point. Just as NSS’ Rick Moy stated on their blog, NSS doesn’t expect that every vendor will necessarily be able to block everything out there (that would assume that innovation and evolution has plateaued in the threat landscape which we know it hasn’t), but they do expect them to deliver on what they assert they are capable of in their literature.
Will Gragido on 12.08.2009
Scott,
Great points. I agree with you and would love to see your thoughts after reading the report. Having said that, I agree with you in that disclosure around security products is not governed like vulnerabilities for example. Having said that, I agree it will be very, very interesting to see how it all shakes out.
Matthew Pour on 12.08.2009
I have been following and buying from NSS Labs since late 2000 when they performed write-ups on Firewalls and IDS’s as a consumer. Now being on the other side of the fence, this is really the only independent tester out there who uses anything near real-world. Being on the other side of the fence now, I continuously try to educate my InfoSec peers on their methodology and experience as not many folks have these attributes.
I concur that quality will improve, but I feel it will only improve with the vendors that have enough engineering pride to move their products forward. God save us if it is a Marketing improvement.
Will Gragido on 12.08.2009
Excellent points Matt. I think they’ll improve if they wish to maintain their positions within the space. For some, if it is not their primary line of business, they may not be so inclined to do so. For others, I would imagine that they’ll have no choice but to do so. I too feel that should it devolve into a marketing evolution the customer base will vote with their feet and move to other vendors whom they perceive to be evolving and pushing forward.
Bob Walder on 12.08.2009
It is important to note when talking about “responsible disclosure” that NSS merely reported metrics in terms of performance and security effectiveness and did NOT reveal exactly which exploits were not detected. Unfortunately it WAS necessary to reveal which evasion techniques were effective against each device. However, given that some of this evasion testing has been on the go since 2001 it beggars belief that some IPS vendors STILL can’t get TCP stream reassembly right!
Rick Moy on 12.08.2009
This test shows why testing is important. Train as you fight. Fight as you train. All live exploits, no neutered POCs.
This test was a long time in the making, and all the tested vendors participated willingly with full knowledge of the test methodology. We look forward to testing again in Q1 for improvements. BTW no vulnerabilities in these systems were probed, just the effectiveness of their stated functionality in terms of blocking attacks against known vulnerabilities. We have been working well with most of the vendors and hope this ‘wake up call’ is answered positively.
The full report can be obtained here: http://nsslabs.com/IPS-2009-Q4
Compliments to the participants of this thread for some good rational discussion. You might find answers to some of the questions on the NSS blog – http://nsslabs.blogspot.com/, and a reply on the NWW article as well where i was misquoted along the lines of expecting 100% protection. http://www.networkworld.com/community/node/48800
Scott Lupfer on 12.08.2009
Bob,
That is my point exactly, I guess I’m just “predicting” that some vendors may say “Well, you should have told us first so we could fix it before you published.” Testing for a vulnerability in a product is different than testing to determine if a product performs as advertised or protects/functions as well as a prospective customer would expect. Two different measurements and two different reporting processes.
This is exactly the type of test that was needed and if I were doing a full and fair evaluation of an IPS, this is exactly the type of information I’d want to make my decision.
If the organizations who didn’t fare well believe that their results shouldn’t have been posted, it is up to them to ensure that their products are up to the challenge.
Great job and I’ll look forward to reading the report.
Scott
PS – The comments here are my own and do not reflect those of my employer or of any organization with which I’m affiliated.
Will Gragido on 12.09.2009
Bob,well said. I think that there are many issues which still require serious investigation and reconsideration such as the TCP stream reassembly issues you described. Rick, thanks for the comment; I think this is a great thread and that we have some great ideas flowing here. Scott, exactly my point; the vendors who didn’t fare well should have been prepared well in advance. That the methodology was posted and the tools well known, suggests that one of three things occurred: they didn’t take it seriously, they prepared against a different set of criteria, there was human error present in some element of their earlier tests.
Kornelius on 12.09.2009
I agree whole hardheartedly!
It is ironic to see how NSS is being featured by the vendors when the results compliment them and then go to war when the case is otherwise. Consumer reporting and investigative reporting should deal with facts and companies like NSS as an example keep the industry they investigate honest.
Often people tend to get lost with the massive marketing campaigns and the things vendors want them to focus on like speed or simplicity while really missing the point of what they are really trying to do. Like snake oil some of those vendor cater to the “pain” customers have by elevating those point while ignoring the very same reason why they came to be in the first place.
NSS would be the consumer reporting I would expect to have for a fair and balanced opinion based on facts not fad or FUD, based in independent testing and not marketing fluff or “Attack CD’s” some of the vendors offer during their evaluation process.
I suggest those companies who scored bad would go and re evaluate their position on the matter and look inwardly and honestly at themselves. I’m certain they would find the real reason for their shortcomings as well as the means to correct them.
I often heard the statement “I’m not going to lose my job for buying Cisco (As an example)” it is my hope that with honest reporting and comparison, those types of statements would be a thing of the past.
NSS should also thank TippingPoint for the vote of confidence…
Will Gragido on 12.09.2009
Kornelius-
Great points! It is totally ironic the way in which those who didn’t fare too well on this latest test are going for the figurative throat of NSS on this matter. Having worked for a few vendors I can tell you all of them have QA deparments and that some take very different approaches to vetting out their technologies. The challenge I think which comes from this report coming out is two fold as we’ve already discussed:
– What needs to be changed within the QA processes of the vendors themselves
– What needs to change within the customers in order to ensure that vendor and product selection meets both business and technological requirements without sacrificing the value which the vendor is marketing to them
I think this will serve as a wake up call for many organizations — be they vendor or consumer. In many respects this ties into much of what my peers and colleagues — here within Cassandra Security and outside of it, are calling for — a challenge to stand and deliver. One can shill for the man selling subservient solutions or one can aim to create and sell solutions which strive to constantly rise above the expectations of themselves, their customers and peers. This is true of anything not just IPSs….
Kornelius on 12.09.2009
Very much so Will,
I would dare to say that those results also reflect those companies understanding of Security, if you will the anatomy of a threat, the difference between an exploit and a vulnerability and how to approach all that while delivering a solution that works in dynamic environments. I’m not entirely convinced that all the vendors featured in the list really “get it”.
Maybe those are Networking companies trying to integrate a half backed Security to their “Cloud” or echo system as they like to call it, sometimes IPS is an afterthought?
Where do they derived most of their revenue? that might be a clue to where they invest the most of their funds.
In one case for sure the focus and attention was pretty much on speed and performance which ultimately made them sacrifice the real essence of why they were created in the first place.
One of the first lessons in computer science is that there are no free meals…
I’m still waiting for the self healing network to come :@)
Will Gragido on 12.09.2009
Excellent points all. You hit on some very strong points here Kornelius. I believe there are varying degrees of comprehension of security at play and at work here. Some vendors have a deeper, more rich and succinct understanding of security than do others. Some have half baked understanding of the space, the problem set, the struggles and challenges while others have a much more focused view. I’d argue that at times security is not an afterthought but a check mark which allows them to play in the game. As for the self healing network, I think we’re all waiting for that….
Billy P on 01.29.2010
First and foremost, I must disclose that I am a Stonesoft employee and our product was involved in this testing. The reason I wanted to reply to this post is because I think it is dead on on how some vendors may not be viewed in the same light, but also to share how different vendors receive and use these reports.
It’s also been interesting to see many other vendors response to these tests. There’s no need to name them here as there comments are public because of their poor results. I can also confirm that this indeed was the first year that NSS Labs did not receive ANY vendor funding from the vendors. In fact, it’s one of the reasons why Stonesoft participated (we already had ICSA Labs certs). This is not to say it had an impact with NSS before, but open it to other vendors.
For Stonesoft, yes, we did very well, in fact amongst the top 3 and beat even some of the largest “market leaders,” however there were areas of improvement and the report did not hold back in mentioning those areas.
These reports serve to offer value for two markets: End users but ALSO for vendors. It’s been interesting to see other vendors rather then embrace the findings and seek for improvement, but suddenly bash these tests and try to promote their own internal results. I can say this, we for one have already begun addressing some of the issues and truly used these findings to help in improving products.
As this article mentions, this may shed some new light on the entire market space. I sure hope so.
Will Gragido on 02.02.2010
Billy,
Thanks for your insightful comment, we appreciate it and look forward to more. I think you hit on some key points and it is quite interesting the gain the perspective of one currently under the employ of a vendor mentioned within the report!
Best,
Will