Recently I wrote a piece on Deep Packet Inspection (DPI), and issues related to its use within the Virgin Media Network in the United Kingdom. In that case, opponents to its use cited UK legislation that prohibits the unauthorized capture of communications and conversations on networks. I found this to be curious as this case dealt with Virgin’s desire to deploy a specific tool that focuses on the detection and identification of peer-to-peer (P2P) networks such as eDonkey, Bit Torrent, and Gnutella. The tool, CView, was to provide a DPI function important to ensuring the integrity and purity of good traffic while aiding in the identification, observation and remediation of bad traffic. Virgin’s concern was the health of its network and its desire to manage anomalous bandwidth consumption while combating illegal use and distribution of copy righted / trade marked materials. This doesn’t sound terribly unreasonable nor does it sound like a ‘violation of privacy’ rights as the opposition suggested.
This case caused me to reflect on challenges and opposition I have seen in hundreds of cases and clients with respect to Deep Packet Inspection (DPI) technologies. I believe many organizations and professionals within them have a fundamental lack of understanding of not only DPI, but also more elementary concepts such as traffic analysis (read: the ability to read and decipher packet captures). I’ve heard every possible technical and cultural argument against the use of packet inspection or session based analysis tools known to man. Some more well articulated and supported than others. I’ve been in countless hundreds of meetings where white board diagramming sessions ensued and network diagram sessions erupted into long debates regarding implementation and positioning of a technology and every reason why a given device could not be implemented. These conversations are healthy and important; they need to be had. However, facts are facts and most of the ‘concerns’ or arguments used against the adoption of intelligent, network-based security products is in fact flawed. Take for example one of my favorite arguments against the adoption of Deep Packet Inspection (DPI) technologies:
- Deep packet inspection solutions introduce latency into the network and in effect are prohibitive to the continued flow of traffic:
- This argument, when first presented, had some validity however DPI solutions, specifically Intrusion Prevention Solution (IPS) appliances have undergone evolution to the third generation. Most if not all offer some form of bypass kit which ensures that in the event of cataclysm (as defined by you or your organization), provided the building is not a smoking crater and that electricity continues to flow so to will your traffic
- Additionally, most platforms if not all offer multiple modes of deployment where in an organization has the ability to segue slowly into full inline integration DPI from a passive monitoring (IDS) mode
- It is true that any time any device is introduced into the flow of traffic some latency – no matter how infinitesimal, will be introduced. This is true of any device be it a router, switch, load balancer, server, firewall, ips etc.
- Deep Packet Inspection (DPI) solutions, specifically Intrusion Prevention Solutions are often never fully implemented often times seeing them remain in a passive monitoring mode. As a result, organization would never fully realize the Return on Investment (ROI), they expected as a result of making their purchase and likely could have settled for a much less sophisticated and costly platform:
- The adoption of the technology and / or the enterprise in questions readiness has no bearing on the efficacy of the technology
- It is intellectually dishonest to assert that had proper due diligence been performed and a readiness assessment undertaken, Return on Investment (ROI) and Total Cost of Ownership (TCO) would not have yielded positive results technical or otherwise
- The threat landscape is moving at a rate which no one can properly contend with and as a result, combat in its entirety:
- This not true for all systems utilizing Deep Packet Inspection (DPI) technology. Yes, there are some which rely on archaic and in some cases less well defined engines and analysis technologies however those who truly ascribe to the definition of Deep Packet Inspection (DPI) should be impacted far less by this than those who do not
- Deep Packet Inspection (DPI) solutions are complex and esoteric; they are not intuitive:
- This argument is weak but needs to be taken in context. In my experience when clients brought this point to the debate table it had more to do with the experience level of their staff than the tools complexity
- This can be overcome quickly and easily provided a proactive, open relationship exists between the vendor and the client
- Education should be ongoing; failure to educate (it is not only the responsibility of the vendor but the organization purchasing and adopting the technology) to ensure
- Fire-walls which adopt and integrate Deep Packet Inspection (DPI) are complex, introduce latency and are less intuitive than their less complex packet-filtering and stateful inspection peers:
- The application of this type of technology is traditionally done by those who are fluent and well versed in the need for it
- It is neither new nor is it beyond comprehension
- Given today’s threats and the complexities researchers and analysts continue to see in record numbers technologies such as this are now more important than ever before not to mention more effective than packet filtering and / or stateful inspection only systems
- Fire-walls which adopt and integrate Deep Packet Inspection (DPI) are complex, introduce latency and are less intuitive than their less complex packet-filtering and stateful inspection peers:
Conclusion:
The reality is that technologies such as Deep Packet Inspection (DPI) are extremely important. Their importance, though debated cannot be ignored and as a result they are more important now than ever than ever before. No longer are we contending with largely unsophisticated threats and adversaries. Cyber-criminals do not sleep. Nor do they take vacations or observe change windows. They are on the job 24 x 7 x 365 and are not deterred by (in fact I would imagine they welcome the objections which are proposed against the adoption of technologies such as DPI) by the presence of traditional mitigative technologies and controls. As a result, the need to strengthen our positions architecturally as well as procedurally must be recognized and acted upon. Time is of the essence and hesitation accompanied by intellectually dishonest or malformed thought processes must be overcome in order to address these threats.
Deep Packet Inspection: A Legal Liability?
Deep packet inspection is not a new concept. It is, in fact, quite mature and takes advantages of the best of IDS (intrusion detection solutions), IPS (Intrusion Prevention Solutions), and Stateful Inspection Firewalls. The technology is extremely effective in combating malicious code and content attacks and in enforcing policy to a variety of ends. Additionally, the technology is quite good at providing detailed intelligence with respect to application behavior and patterns as they appear within a given infrastructure. In modern enterprise and carrier networks this technology is both common and integral in ensuring operational efficiency while managing and minimizing risk.
Recently it has come under fire however and in at least one case, been dubbed a measure by which the privacy rights of end users can and will no doubt be violated. The case in question is that of the recent announcement by Virgin Media to utilize and deploy a DPI like technology package called CView within its network environment in order to better understand the prevalence and associated patterns of use seen in peer-to-peer networking sessions. The tool would be in effect, capable of tracking sessions associated with peer-to-peer networks such as Gnutella, Bit Torrent or eDonkey which has created a negative buzz amongst organizations such as Privacy International who appealed to the EU to step in and review the package proposed by Virgin Media. Virgin’s intentions seem straightforward to me but perhaps that is due to my being an information security professional:
- Gain an understanding of the usage and patterns of associated usage with these P2P networks and clients
- Analyze instability presented by them within the network in terms of inordinate resource consumption
- Analyze content for purposes of legality (avoid in the trafficking of either copy righted material of illicit illegal material)
- Implement throttling if necessary
- Implement policy control if found to be necessary by law or by virtue of Virgin Policy
- Mitigate risk posed to the Virgin Media network environment and its user community
- Prevent malicious code and content propagation to and including the propagation of advanced malcode kits and bot nets
I have to believe the goal of using a tool such as CView (if you look the tool up you will see it does not tie individual identity information to information harvested) is pretty straightforward and reflects much, if not all of what is seen above. I find it hard to believe that this is a case where privacy should be an issue though I am aware that in the UK under the Regulation of Investigatory Powers Act (RIPA), intercepting communications is a criminal offense regardless of what is being done with the data. While I am no expert in British Parliamentary process or law, it would seem that this act would be prohibitive, if not crippling in providing advanced security solutions while potentially curtailing illicit, illegal activity. Deep packet inspection is not the problem here, the problem is perception as it relates to the lengths to which personal ‘freedom’ extends and illegal activity begins.
One of my favorite parts of penetration testing is and always has been social engineering. I love it. In fact, I love it so much; I developed a real passion for it. My passion led well beyond the traditional social engineering and evolved into the study and practice of more sophisticated techniques associated with socio-psycho manipulation. It is a gift of sorts and who am I to question a gift? When speaking with prospective or current clients with respect to security assessments I have often implored them to include both physical security analysis and social engineering. This habit was formed upon entry into the world of professional security consulting where I sharpened these skills under the tutelage of many senior to me in both age and experience. Almost all of my earliest mentors in this space were or had been active in the United States DoD or like intelligence services, most of whom shared a common pedigree not unlike my own hailing from the world of information security and intelligence. These environments taught us to remain vigilant and open to idea that the unexpected should be expected therefore our ability to address anomalous events must remain categorically strong.
Over the years, within many engagements, activities of this nature were written into SoWs (Statements of Work for the lay person), and acted upon with full approval of parties both able to authorize such activity within and against their organizations and their legal representation (a detail I cannot stress enough that should not be overlooked). We would engage reconnaissance carefully leveraging skills we had cultivated in our former lives and applying them to the commercial world. We would engage in deep open source intelligence gathering and analysis in order to supplement our knowledge base regarding our target(s). We would become familiar with the physical environment in which our targets could and would likely be found. These locations would range from their places of business, to local restaurants and pubs, to local shopping districts to other geographic locales that we knew to be frequented by parties belonging to the organization, our target, in question. All the while we would be looking for ingress and egress points in addition to ideal areas for initial contact and exploitation. We would additionally identified areas in which useful information may or not be discarded by our targets that truly believed that no one would be interested (ah how I loved dumpster diving in my youth!). Finally, upon having enough information we would begin our careful insertion and infiltration. There is an art to doing this as much as there is a science and at times, the art becomes (or at least in my experience became), much more valuable.
These exercises almost always proved fruitful, as they would typically be multi-faceted involving multiple small teams each of which was cognizant of the others activities but all tasked with different missions during these exercises within the greater assessment. Some teams focused more upon the physical compromise of the environment, the acquisition of credentials and knowledge which could be socially engineered or stolen and thus counterfeited to serve our purposes in pushing deeper into the environment escalating our mission per our SoW and charter. Others would work on compromising individuals, seeking to gain their trust and subsequently their knowledge and any information considered noteworthy. Others still work on electronic and social engineering utilizing e-mail, web and telephonic techniques all designed to gain the trust of our targets or members of our target organizations in the hopes of escalating our privileges and advancing our efforts. This was good work. It was important work. And it was work that not all are capable of nor designed for. To the layperson reading this post I imagine it sounds outrageous if not frightening that work of this nature goes on and is conducted by security professionals in an effort to test and assess the security posture of a client and to a degree I can understand that attitude. However, this work is terribly important and often demonstrates weaknesses not previously accounted for within enterprise environments (public or private), during traditional security assessments and audits.
At this point you may be wondering whether or not there is merit in engaging a qualified team to do provide this type of service in addition to traditional services brought to the table as part of a security assessment. My personal perspective is that if you have overt responsible for the risk posture of an organization and understand that security or the state of being secure is contingent upon the three legs of the security stool (people, process and technology) being dutifully tested and exercised you most assuredly should do so. Failure to do so can expose you and your organization to a world of risk, which complying mindlessly with a three lettered data security standard or a mutagenic health-privacy act cannot hope to save you from. So what are we to do? First, if you haven’t already done so, conduct a meticulous review and analysis of your organizational information and physical security policies. If you don’t have any now is the time to remedy this deficiency. Should you (as I imagine most do, but will refrain from assuming), have these policies on hand review while employing a healthy dose of third party ego; remove yourself from the immediate, intimate involvement with them as they relate to your job and evaluate them as though you were brought in to do so as a third party. Do they look mature? Are they clearly articulated and well defined? Are they comprehensive? Do they address the natural bridges that occur between physical and logical security? Provided you have the resources, engage them to conduct preliminary assessments (provided they are qualified to do so), and if they are found to be lacking in the ability, do not hesitate to contact professionals with the appropriate pedigree, background and reputation to speak to about scoping a statement of work on your behalf. Remember that not every attack of a truly serious nature takes place across a wire and that many begin and end with a simple telephone call, a conversation around a smoking area or via a new hire.
Determinism vs Randomness: The Net Effect Upon Security
By nature, I am an empiricist; it is who I am and works for me based on my bent toward analytics and multi-faceted (at times onerous), levels of thought and pontification. I am unapologetic about the way I approach things; it is simply who I am. Having said that, I recognize that I am not – nor is my way of approaching things, universally embraced or right for everyone. To assert otherwise would be intellectually dishonest. I am particularly intrigued (and spend a lot of time reading and studying), determinism and randomness theory and philosophy. For many of us, life is as simple as asking a question which the quintessential Canadian thinking mans band Rush asked on its 1991 album Roll The Bones “why are here, because we’re here, roll the bones”; while for others the question of why and perhaps more importantly the answer is not so simple. I fall into the latter camp.
I a student of empiricism; I am a stalwart advocate of critical thinking and reasoning especially when it deals with philosophical schools of thought such as determinism vs. randomness and how they interact within the world in which I professionally live and work. These ideas are not new. In fact they are quite old. They are in many respects extremely old and as a result of their vintage, they have been and remain the subject of great debate. Authors and thinkers such as Nassim Nicholas Taleb, who wrote two of my favorite books on the subject : Fooled by Randomness and The Black Swan: The Impact of the Highly Improbable, go to great lengths to explain these concepts along with their impact on causality. So too did David Hume, the famed Scottish philosopher, along with Karl Popper and Colin Howson. Needless to say there is a long and strong tradition in examining deterministic vs. random philosophy as it relates to probability. The concepts are as old as time itself; as long as mankind has had the ability to reason he has struggled with whether or not events occur due to deterministic causes (or more appropriately because of events which exist and influence other events thus arriving at the cause for a current event), or due to sheer randomness. We are no different than our predecessor in this respect. We seek knowledge with respect to the origins of things and events in addition to what there existence will mean to us as we move forward. This desire to know unequivocally what influences outcomes and the probability of those outcomes is central to the theme of our existence. As a result, it infiltrates (if we are paying attention), all aspects of our lies from the most complex to the least. We find ourselves asking why certain things occur at the time and place that they did, and to what end. I happened to be in New York City last weekend making my way to LaGuardia Airport via the Holland Tunnel at the height of the melee that was underway surrounding the events of the car bomb discovered in Times Square. Needless to say, traffic through the Holland Tunnel neither was less than forgiving nor was that which we encountered on way to Queens any better as a result. On the trip into the city news commentators could be heard speculating with respect to the cause of this event. Why would a young, respected young naturalized American citizen (Faizal Shahzad), find it acceptable to place a makeshift bomb in Times Square? What was his reasoning? His goal? His message? Who was behind the activity and what might be the logical extension seen as a result of this event? All valid questions. All seeking validation with respect to understanding whether or not the causality associated with these questions and the event in question (not to mention the young man), was in fact deterministic in origin or random. We know that it was in fact not random based on evidence that had been collected and authorities are continuing to investigate the events that lead to this event and ultimately influenced it from the perspective of cause. We humans tend to this with all manner of things ranging from the serious to the trivial.
With respect to information security or security in general, I believe we do so more often than people realize. Security or being secure, is in many respects dependent upon being able to detect, identify and observer causality. In being able to accomplish these three things, we are better positioned to account and prepare for the unknown. If you stop to think about that for a moment it should become quite clear that the act of securing anything – home, car, host, server, network, people – requires the acknowledgment of historic reasoning (in both deterministic philosophy and randomness), while at the same time the acknowledgment of the unknown.
We see this often within the friendly confines of our industry. Take for example the following: An organization is instructed by a governing body that in order to achieve a state of conformity with its governing body the organization in question must meet and demonstrate achievement of x number of criteria. Failure to do so will result in negative ratings that may or may not result in fines and / or the inability to conduct business transactions. The governing body assumes that arriving at a state found to be in alignment with its standards will discount and eliminate (due to deterministic causality), any potential for randomness to manifest, thus negating the possibility. But what if their assumption is wrong? What if the data which they have assumed to be whole and comprehensive is not so?
I fear that this is more common than not within our space due to a lack of due diligence and grasp of historical accuracy with a forensic like precision.
Here’s another example:
A software-publishing house for quick processing of financial transactions develops an application. It is seen as being mission critical to organizations that purchase it looking to capitalize off of any edge they can to beat their competitors to the market. Speed in this case is very good. The software publishers, realizing the importance and value of the application to their clientele decide to expeditiously develop and push the code to market rushing through all quality assurance (QA) and beta testing in order to beat the deadlines set by the executive teams in order to realize the greatest degree of revenue possible. The developers run through the exercise of white boarding the data flow and block diagrams, technical requirement documentation, marketing requirement documents and product roadmap documents. From there the code is pushed through the QA gauntlet at light speed and rushed into the beta testing customer environments. Initial results are noted and brought back to product management and engineering who then wrestle with addressing the issues in a timely fashion in order stay within budget (both financial and time budgets), while not missing their window of opportunity within the market space. The code is run through QA again, and pushed for GA candidacy.
But there is a fly in the ointment. Some young (or not so young), perhaps charismatic (or at the very least quirky), individual is asked to look at the code or application as part of an audit and assessment and finds that low and behold it is vulnerable to an abundance of potential threats all of which can be exploited in a trivial manner. At the same time this assessment is occurring the code and its publishers are reaping great successes and accolades. The code, now a fully baked financial suite is swiftly on its way to becoming one of the most popular suites of its kind in 21st century business; yet, it is as vulnerable to exploitation as a runaway at a Port Authority bus station. While our young or not so young, assessor of questionable charismatic quality, is reviewing the code, carefully noting the deficiencies and potential for complete exploitation, reports begin trickling into our software publisher that exploitative events have begun. Worse yet, they were events that were not accounted for during initial or secondary quality assurance testing and thus perceived as being random. We know however that randomness is simply the failure to take note of events that feed into causality, which therefore can be interpreted as a failure in paying attention to detail. Perhaps one of the gravest mistakes anyone can make yet all too common within our world and history, let alone our industry. So what are we to do about this? How can we, as professionals convey a sense of urgency that supersedes and avoids a “chicken little” like knee-jerk response to events we encounter? This is easier said than done especially in a world where information travels at the speed of light. I believe that in order to achieve the proper perspective we need to encourage the following:
- a healthy respect for that which is known or what we know to be true
- a healthy respect for the known or what we are not sure of
- an acceptance of our current posture as we understand it
- a recognition of our strengths as we are aware of them
- a recognition of our weaknesses of weaknesses as opposed to a denial of them
- an ability to process this information and begin formulating a plan
- The ability to execute that plan and due course perpetuate its repetition in order to avoid falling victim to said trappings.
This is by no means a trivial event; nor has it ever been an easy proposition. The ability to interpret historical events and data — even when they appear to be disparate and unrelated is paramount to achieving the goal of comprehensive deterministic understanding. In short this allows us to avoid via scientific means the pitfalls associated with randomness and its associated theories. In order that we may achieve this the ability to reflect upon our data sets and circumstance all while applying observing ego is of paramount importance.
New White Paper: The Rise of the Cyber Cell
The Evolution of a Sub-national Entity in CyberSpace: The Rise of the Cyber Cell
Security Awareness: Hope At the Bottom of Pandora’s Jar
Recently I spoke at a private conference sponsored by a global, multi-national manufacturing and biomedical organization. It was a real pleasure to speak there for me as I was doing so with a colleague and it is always fun for me to present in such a way. The topic for our presentation was influenced by information we received from the organizers with respect to their wide and diverse audience, an audience which during the initial presentation would include 130 + people in person, and several watching via streamed video in 7 different countries. It would eventually be used by the organization to educate their 26,000 + day-to-day computer users, something to be pretty excited about. These users, like users in many other organizations ranged in experience level, some having basic knowledge of information security and others having much more in-depth experience. It was going to be a fun presentation and the opportunity to share knowledge and, in turn, be exposed to the experiences of others was going to be worth the effort.
Realizing the diversity of the audience and experience levels, we decided to produce a deck which would explore the Internet Threat Landscape touching on key ideas and concepts which the organizers believed to be appropriate for the time and audience. We were to speak at the end of the day so we decided to encourage an interactive approach during our presentation versus the traditional academic style presentation. We were privileged to wonderful audience response as a result. A calculated risk but a risk nonetheless. A lot of information was covered in a relatively short period (we lost 30 minutes of a 90 minute slot and were notified not long before taking the stage). Lights, camera, revolution! Opening remarks were made, obligatory joke to break the ice and set the tone, and then through the looking glass and on to the Threat Landscape. It was going to be a good presentation. As we progressed through the deck it occurred to me that there were looks of disbelief, shock and awe appearing on the faces of many in the audience. They dotted the local landscape the way wildflowers do hillsides in Spring. Additionally, the knowing nod of heads could be seen as well; a good sign that the mark was being hit. As the presentation continued to flow we began introducing common threat vectors being exploited along with a brief historical overview of malware from 1971 to the present. I introduced the idea of evolution occurring naturally within the sub-ecosystems and greater ecosystem which accounts for the ecology of the Threat Landscape. It occurred to me while introducing this idea to the crowd how pedestrian things become when you are exposed through research, analysis and extensive study to your subject matter and yet, how powerful and enlightening something can be to fresh eyes. It is part of what we do which I love most; the education and subsequent recognition of the new. It is a beautiful thing.
We introduced the concept of web based application attacks, and though we didn’t have time to provide any real-time examples, demonstrated statistics provided through our own research and that conducted by organizations like Whitehat Security, Inc. and IBM ISS X-Force. These statistics spoke to the prevalence of vulnerabilities such as SQL Injection, Cross Site Scripting (XSS), and Click Fraud; specifically their commonality in Internet based application activity seen today. Some of the statistics were shocking to the audience; you could see it in their eyes; they we not prepared to hear them; to see them; to realize what they meant to them on a personal level. Next, we introduced the concept of cyber-crime to the audience and began discussing just why someone might go through the lengths required to exploit one or more of these vulnerabilities. As the realities associated with our topic matter began resonating with the audience, again looks of disbelief appeared on the faces of some; sometimes only appearing in their eyes, while knowing looks appeared on the faces of others as they nodded their heads in agreement. This too was good.
We discussed the role of the individual operator, broached on the concept of confederations and criminal exchanges and then touched briefly upon the role of true organized crime entities in this space. In order to the drive the concept home we elected to introduce the concept of the botnet to the audience in order to illustrate some of the points we were making with respect to our topic matter. I spoke about botnet architecture, the role of cryptovirology in hiding binaries (making them undetectable by signature solutions and non-signature solutions alike), and much more. It felt good. What didn’t feel good was the realization that there is so much more education to do and how often there is so little time to do it within in order to be effective. This organization was no different than many that I have spoken at or consulted with over the years. Fortune 1000 organizations often times have the same problems as do Fortune 50 organizations. Some organizations embrace education and awareness more seriously than do others while there are some, who in order to protect the identities of those who work and toil on a daily basis in an effort to try introducing change, shall and need to remain nameless. Shattering the illusion of security via obscurity is as important within these as any other, perhaps more so. I encourage more education of this type. I believe there has never been a more appropriate time for it. The advent of Web 2.0, mobility, universal connectivity (all topics touched on in the presentation provided at the conference mentioned above), affects us all in both wonderful and potentially dangerous ways. It’s a situation akin to Pandora’s Jar where upon opening it, much danger; much evil was released into the world yet when finally exhausted, at the bottom of the box, there lay hope. We must have hope. We need to encourage this and in encouraging it, we can encourage change. Much needed change.
