Here is some of the recent press coverage on topics relevant to content on my blog:

• Information Week: Virtualization Has A Security Blind Spot
• Information Week: Securing Virtualization, or is that Virtualizing Security?
• Network World: Black Hat speakers expose virtualization, OS security gaps (**NOTE: Please see here, VERY important)
• Network World/Computerworld: Black Hat spotlights virtualization, DNS issues (**NOTE: Please see here, VERY important)
• SearchSecurity (Australia): Could securing virtualised environments destroy ROI?
• SearchSecurity: Initial virtualization costs could outweigh benefits
• Computer Zeitung: Today’s Security Products Aren’t Ready For Virtualised Data Centres
• Wall Street Journal: Hackers On the Move
• Baseline: Managing Mobility In the Enterprise
• ITWorld: Pros and Cons of VMware’s New Security Guide

Podcasts/Webcasts/Video:

• Network Security Podcast: Defcon Wrap-up (*naughty language)
• Security Wire Weekly: Virtual Security Apocalypse
• DarkReading: Four Horsemen Interview after my talk at Blackhat
• SearchSecurity "Still Nameless Podcast": Rich Mogull and associated nonsense (I wasn’t on it, just in it 😉

I am confirmed to  speak at the following upcoming events:

• Forrester’s Security Forum, Boston MA, September 4-5
• [Private] Portland, Oregon, September 8-11
• [Private], Utah, October 1-2
• Day-Con II, Dayton, Ohio, October 10-12
• RSA Europe, London UK, October 27-29
• TechTarget Information Security Decisions, Chicago IL, November 5

I will be attending the following events:

• VMworld 2008, Las Vegas NV, September 14-18

/Hoff

What the hell’s goin’ on here?
something’s surely a mess,
our BGP is announcing
the wrong damned AS

See, I announce with this prefix,
it’s a slash 24,
here to there should take 3 hops,
not 18 or more

I’m pinging the next hop and
that works just fine,
ping a host, subnet over,
slows like a POTS line

That Defcon session,
when we IM’d all night,
that shit’s all encrypted
you told me that, right?

My telnet shell’s cleartext!
DONE! Stabbed it with a FIN fork
So why do these Pcap’s
show SYN’s to New York!?

Somethin’ sure does look fishy,
TTLs all askew
are the ISPs tapping traffic
‘tween me and you?

I’m just paranoid, man,
I’m sure it’s all fine.
These ping-pong effects?
BGP’s grand design

I mean really, why worry?
Even though, I confess,
it’s not like we’re vulnerable
like with DNS

BGP must be foolproof
auth’d and encrypted
there’s no way they’ve gamed it,
redirected or sniffed it

It would be quite stupid
if AS routes, you could twiddle,
intercept all my traffic
with a man-in-the-middle

Nah, I’ll sit here, use torrents,
my bits are secure,
close my eyes and imagine
that the Internet’s pure

What’s next though, I wonder,
what protocol hack
will cause Internet chaos
and make the tubes crack?

I was blipping through my RSS reader this evening and noticed this new little doozy of a headline referencing a story that is now weeks old:

Revealed: The Internet’s Biggest Security Hole

Holy crap! That’s pretty scary looking, huh?  Another Internet’s biggest security hole!?  I can’t take another.  I don’t have another poem in me.  What sort of "fool" disclosure is this!?

Then again, there are plenty of big ‘holes on the Internet, so I thought I better make sure it wasn’t me this time 😉

Kapela’s and Pilosov’s cool performance at Defcon was sadly drowned out by Uncle Dan’s DNS flaw and the sheer weight of his grandma’s cookies (which I received zero samples of, by the way ;( )

The gist of this story is that by utilizing the built-in friendliness of BGP, you can cause bad things™ to happen by redirecting, intercepting and then sending traffic back on its way with a high likelihood of not being detected.

"We’re not doing anything out of the ordinary," Kapela told Wired.com.
"There’s no vulnerabilities, no protocol errors, there are no software
problems. The problem arises (from) the level of interconnectivity
that’s needed to maintain this mess, to keep it all working."

It’s another case of "everyone knows this can (and probably does) happen, but we’re just hoping it doesn’t," and very smart people have been warning others about this for years.  You shouldn’t drink the water overseas, either.

Even as recently as the YouTube/Pakistan issue which was a BGP-related issue that caused a DoS, not-so-smart people such as your humble author suggested exactly this sort of thing was possible:

Yes, this is really a demonstration of unavailability, but
what I’m getting at here is that fundamentally, the core routing
protocol we depend upon for the backbone Internet transport is roughly
governed by the same rules that we depend upon whilst driving down a
road separated by nothing more than painted lines…you simply
hope/trust that nobody crosses the line and crashes into you head-on.

There is very little preventing someone from re-routing traffic.
This could result in either a denial of service (as the traffic would
not reach its destination) or even something akin to an interception,
"storage" and eventual forwarding for nefarious means.

So, here we have a case where again we depend upon a protocol that
was designed to provide (A)vailability, yet C and I are left
floundering in the wings.  We’ll no doubt see another round of folks
who will try and evangelize the need for secure BGP — just like secure
DNS, secure SMTP, secure…

This will hit deaf ears until we see the same thing happen again…

Ooooh.  I must be psychic.

Wait until I demonstrate how to redirect the NetBIOS traffic of every Win2K/XP box that has NBT bound to the NICs by a cleverly devious combination of ICMP source quench, token ring beacons and uPnP.

I’ll be FAMOUS!

/Hoff

A point I’ve raised in my Four Horsemen presentation that I’m spending a lot of time researching is on the topic of how virtualization is accelerating the adoption of SAN storage and to a larger extent I/O virtualization and what that means to security.

I wanted to introduce this topic as a teaser to a larger series of posts.  This is not a treatise on the subject and is deliberately thin on detail and thick on corner case illustration.  You have been warned.

What I’m both extremely intrigued by and really worried about as we move further along the virtualization continuum is the strange duality offered up by the intersection of consolidated compute, resources and information overlaid with tremendous distributed mobility and the security (or lack thereof) of this pooled storage holding all our crown jewels.

Centralized storage offers great benefits: easier to backup/restore, simpler management, better cost effectiveness, facilitated resiliency and potentially better security.  However, that all depends upon how and who is actually responsible for the operational aspects of security and defining the policies and compliance requirements in the first place.

We see the grappling matches of responsibility being waged between who is responsible for securing our basic virtualized environments from the "server" and "network" perspective today and we’re struggling for answers.

What about securing storage?

This isn’t an argument about filesystem partitioning, ACL’s and GPO.  This is a whole other networking fabric or set of appliances that are being deployed, conntected to our hosts and administered/secured by…who?

Think about it; we’re moving away from local storage to pooled "networked" storage.  Not only is our critical information stored in these pools, but the Virtual Machine (VM) images are also.  Databases too.  One stop shopping!

Sure, this has trend has been going on for years, but virtualization and consolidation are shining the ugly light on the fact that we’ve been covering our eyes as to what this means to our overall security posture for many years.  That’s not going to last much longer.

Depending upon who is responsible for the architecture of your virtualized storage, your perfectly reasonable asset, network segmentation and layered defense may go out the window when "machines" from multiple tiers all interconnect to a single storage fabric. 

It shouldn’t happen, but it does; think about your (coming soon) virtualized DMZ’s and what that might mean to this diagram:

You might notice that for "simplicity" although the management/service console network is represented, the storage "network" is not.  I’ll bet dollars to doughnuts that all three tiers would be serviced by a single SAN in the real world…

I’ve heard that over your dead body would you combine multiple network zones on the same physical/virtualized host like in the picture above.  Strangely though I bet many of you connect physical assets (virtualized or not) of varying criticality to the same SAN fabric though like in the Brocade diagram on the top left.

What exactly is the difference?

Will the real SAN storage security experts please stand up?  OK, how
about if you’re faking it really well?  Um, how about you storage
admins posing as security experts?  Server admins who eat LUNs for
lunch?  Network engineers who admin Cisco, Brocade, Xsigo I/O virtualization switches?

Bueller?

I am *not* a security storage expert but I’m reasonably skilled in many
other elements of security, and that’s really the point of all of
this.  I recognize there are many things that can be done to segment/zone/mask/secure
storage, but I don’t have those skills and I don’t know how to assess others’ either.  I’d bet that if you haven’t
been involved in securing storage for quite some time, even if you’re
being drug into virtualized environments, you’re in the same boat?

Food for (scary) thought.

/Hoff

UPDATE: Here’s the latest version of the presentation as I updated it for SecTor in Canada.  It includes many additions as well as modified single slides of the animated ones.

You can find the slides from October 2008 here.

There were some significant differences in the slides that were on the CD issued by the Blackhat folks and what I delivered.

You might be interested in them.  I’ve exported the presentation to PDF with each animation built as a separate slide – in some cases that means there are 5-6 slides with advancing bullets, graphics, etc.  As annoying as that may be, it fixes the mess of the positional overlay problem you’ll see if you view the PDF from the CD.

As an important note, my slides are designed to accompany my speaking, not the other way around, so in some cases they don’t explain themselves.  This is by design 😉

I will be giving updates to this presentation throughout the rest of the year since it’s a presentation designed to communicate the virtualization “state of the art” as it relates to VirtSec.  So, if you attend a conference and see this talk advertised, it will have new/updated content.

Be warned, this PDF is huge (~55 MB) because my slides are intensely graphical.

Enjoy.

/Hoff

(In 5 days there have been almost 1000 downloads of this preso.  If any of you have feedback, I’d really appreciate it.  Thanks)

One of my key topics in my Blackhat presentation "The Four Horsemen of the Virtualization Security Apocalypse" focused on today’s lack of state-synchronized high availability/load-balanced solutions for security virtual appliances that offer the same functionality as their physical appliance counterparts.

I used an example from a vendor I didn’t name in one of my slides to illustrate that if you attempted to replicate the options you have today in physical appliances in a virtual construct, you would not be able to (using this particular solution suite as an example.)

That vendor was Check Point and I was referring to their SPLAT-based virtual appliance. 

The version that was available for ESX environments when I created my presentation offered no load-balanced HA capabilities whatsoever as stated in the release notes. 

However, today I received the announcement of Check Point’s VPN-1 Virtual Edition (virtual appliance.)

Based upon what I am reading in the administrator’s guide, VE now includes support for ClusterXL HA FW/VPN state-synchronized load sharing across one or more ESX hosts.

This is a great first step in the right direction.

We should expect more solutions to start arriving shortly to address many of the issues I brought up as the current "state of the art," and this is a good example of that. 

There are numerous caveats and limitations, many of which I spoke directly about in my presentation with many of them related to the interdependencies of network topologies, virtual networking and interaction with VMware’s integrated HA/clustering solutions…which was one of the other key topics in my talk 😉 I’ll update some of them as an addendum to this post shortly. Updated below.

You can find information regarding Check Point’s VPN-1 Virtual Edition here.

/Hoff

Updated: If you read the release notes, you’ll notice these little gems.  I don’t have the time to go through each of these limitations, but they’re significant and go to the point that all things are not equal in V versus P:

• Interface bonding on the virtual machine running the VPN-1 Virtual Edition is not supported
  with ClusterXL.
• VMotion is not supported
• VPN-1 gateways in the Bridge Mode must have their internal and external interfaces connected
  to port groups that are configured in promiscuous mode.
• VPN-1 gateways in the Bridge Mode are not supported with ClusterXL.
• Virtual machines may be connected to a maximum of four different virtual switches. This may
  limit the number of virtual networks protected by a VPN-1 Virtual Edition machine. This
  limitation can be overcome using VLANs.

I’ll give them props for this one, though, given its refreshing honesty:

• VPN-1 Virtual Edition does not protect the VMkernel.

I just responded to a comment from Iben Rodriguez on one of my virtualization and PCI blog entries from a while back and posted an observation while at the same time managed to make a funny (see the title.)

I wanted to both reflect upon Iben’s comment as well as chuckle a bit.

From what I extracted from his comment, Iben is suggesting that perhaps virtualization should not affect an auditor’s approach or differentiate the audit process from a physical server depending upon the definition of a "server:"

Is an ESX Host a server?

It should be considered similar to the chassis holding a bunch of blade servers. 

These have management ports on separate networks, with LDAP authentication over security protocols like ssh and ssl.

And why not treat them as a hybrid device with different network switches, storage controllers, etc?

Vmware has recently removed the word "Server" from after the ESX product name…

It’s not a server, it’s a hypervisor.

It’s not a server, it’s a switch.

By defining what a server is and is not a PCI Audit should be pretty straight forward.

I think this is a messy question and one we ought to continue to address.  I need to go and check out my ISACA references to seek guidance on this matter from a, um, "higher" source 😉 I do think that ultimately this is a very subjective issue, to which I responded:

The answers to your questions/suppositions are quite simple:

"It all depends upon the auditor."

Most of the folks I’ve spoken to recently are essentially counting
upon the ignorance of the auditors and the general confusion regarding
terminology and technology to glide by at this point.

Server/blade/hypervisor/switch … it’s all fun and games until someone loses a (PC)I… 😉

"As long as I put in place the same host controls I do in a physical
environment and not tell the auditor it’s virtualized, it’s all good
and what they don’t know, won’t hurt me."

Sad but true.

From the Department Of Serendipity…

Thanks to a heads-up from my buddy Jack Daniel, rumors that Leo Laporte read my DNS Debacle poem on the Security Now podcast are confirmed. 

I’ve listened to and watched Leo’s shows for a long time and it was very cool to hear him rattle off my prose. 

Despite the glorious buzz of two-stroke gardening equipment in the background, Leo’s fantastic radio voice, dripping in the style of Dr. Suess, added a surreal quality to my poem as he read it.

Thanks very much to both Steve Gibson and Leo Laporte for the nod.

It starts at around 18:40 into the podcast.  Check it out here.

/Hoff

The resiliency and availability of security appliances in virtual environments is a focus of my "Four Horsemen of the Virtualization Security Apocalypse" presentation which I delivered during Blackhat last week.

In my session I discussed some detailed scenarios of the architectural adjustments to infrastructure that virtualizing physical security appliances require and what that means to the overall resiliency,
reliability, performance and scalability of systems which depend upon security controls in the form of physical appliances today.

Specifically, I highlighted some of the limitations of using security virtual appliances and demonstrated how relying upon virtual security appliances can actually decrease the security posture and increase risk in our environments today.

One of my examples illustrated how it may become necessary to combine multiple security virtual appliances on a cluster separate from the VM’s they are protecting in order to achieve the availability, performance, scalability, and resiliency that we get out of dedicated in-line physical appliances today.

If you prefer a simpler example, I also presented the simpler example of a firewall virtual appliance front-ending 10 production VM’s.  I like the former because it directly contrasts the physical and completely virtual.

For the sake of illustrating a point, imagine if one were actually able to satisfy business, security and compliance requirements using this virtualized security architecture in a production environment built upon the same virtualization platform as non-security guests.*

Not to pick on my friends at VMware, but today’s license timebomb issue delivered by a VMware patch is pretty nasty and sends my hackles skyward when contemplating what it would mean were one to virtualize the security infrastructure. 

Basically, this issue caused by an update rendered certain VMs unusable after the update.  If one or more VM’s on an updated host happened to be a security appliance or security control taken down for patching or rotated for re-purposing, etc. imagine your surprise post-patch.

Remember that security applications are very topology and state-sensitive and
unlike other apps. that just care about an IP address with which to spew packets
ethereally, security appliances/applications need to bind access
policies with affinity in order to protect assets behind them.  As we
all know, when something doesn’t work, we invoke the SysAdmin prime
directive: "Blame the firewall!" 😉

Events such as this may cause some to give pause to enterprise security architects before migrating security functions to virtual appliances, especially given where we are today with what I presented in terms of high-availability options within single-cluster hosts with virtual security appliances. 

In reality, it will probably cause people to consider what virtualization means as an overall contributor to operational risk for any physical system conversion — regardless of vendor — since any VA/VM would be affected, but let’s think about this as if one had virtualized the security infrastructure.

While it’s true that for guests we have DR options and snapshotting that would make roll backs an easy affair, this shines a spotlight on the difficulties with patching the underlying virtualization platform and what that means to operational resilience.

The notion of a homogeneous virtualization platform are certainly compelling; easier administration, patching, configuration, standardization, reduced costs, etc.  However, the notion of a monoculture "operating system" has its downfalls also.  This issue clearly highlights one of them.

I’m not suggesting that there are not opportunities for virtualizing certain security functions, but as I pointed out in my talk, many of the required topologies and high-availability options present in mature physical security appliances are not available in the virtual appliance world.

Today’s issue highlights the need for very careful planning when comes to what, when and if one should
virtualize security functions.

When in doubt, refer to Hoff’s Corollary:

/Hoff

* You might want to look at what a platform like the Crossbeam X-Series can give you that "normal" virtualization security platforms cannot, as it mitigates some of the issues mentioned in my talk.

** Of relevance is my blog post from back in January titled "On Patch Tuesdays For Virtualization Platforms"

Based upon feedback from attendees at Blackhat, my talk, "The Four Horsemen of the
Virtualization Security Apocalypse," went over well and I really had a lot of
fun delivering it. It’s had a TON of coverage.

Despite the positive feedback from folks, it seems the foreboding narrative of the apocalypse has carried over into the real world due to a rather unfortunate journalistic misinterpretation of the facts.

It’s only fair to state that I have been critical in the past of others in our line of work who have complained of their inability to control the output of their direct interviews with the press and analysts as misquotes and misunderstandings arise.

Perhaps this is a little karmic payback for my outspokenness, as after my talk at Blackhat, I have now enjoyed the fruits of journalistic distortion firsthand.  It’s important to note that this was not the result of a direct interview, but rather the inaccurate reporting of a reporter sitting in the audience of my talk.  I was never contacted with questions or asked for clarification or review.

Many of the points I made in my presentation were reflected upon poorly and my perspective butchered, but one specific item is causing me some serious grief in a professional capacity.  It cast a rather crappy pall on the rest of my Blackhat and Defcon experience (more on that later.)

One of the "Four Horsemen" which represents a critical issue in virtualization security is that of the hidden costs involved in virtualizing security.  The point I made, and the language I used to consistently describe it multiple times appears below:


To be perfectly clear, what I obviously said was that "virtualizing security will not save you money, it will cost you more."

What Ellen Messmer reported in her Network World article was that I said "Virtualization will not save you money, it will cost you more.”

Now, this may not seem like much of a difference, but it’s a profoundly impacting dissimilarity.

It’s a dangerous rephrase that has now caused significant pain for me that I’m going to have to deal with once I return from vacation.  It’s been picked up and re-printed/adapted so many times without validation that I can’t keep count any longer.

You see, I work as the security architect for the division of a company who is maniacally focused on designing, deploying and supporting heavily-virtualized realtime infrastructure for our customers.  One of the (obvious) value propositions of virtualization/RTI is cost savings/reduction/avoidance which I specifically referenced during my presentation as a well-established fact and reasonable motivation for virtualization.

You can probably imagine the surprise of folks when they read Ellen’s article which is written in a way that directly contradicts our corporate messaging and the value proposition offered to our clients.  It reflects rather poorly on me and my company.

And just to be clear, my scorn was not directed at the "network industry" or the "virtualization industry" as reported in the article; the context of my entire talk was the security industry, a point sorely missed.

This article reads like the output result of a bad game of "telephone."

I intend to contact Ellen Messmer and ask for a retraction as well as corrections of multiple other mistakes in the article, but as we all know, there’s no real retraction on the Internet.  All I can offer is my presentation, the video recording of it and the recollection of the 500+ others that were in the audience when I presented (including numerous other reporters.) 

The only other thing left to do is to sheepishly admit that despite the fact that this was not an interview that I or anyone else could control or influence for correctness, Joanna Rutkowska was essentially correct in her assertion during our last debate that you cannot control the press, despite best efforts. 

Even though I’ve never had a problem of this degree in the almost 15 years of doing this sort of thing, I humbly submit to her on that point.

/Hoff