Rational Survivability

The topic of security awareness training has floated up to the surface
on a number of related topics lately and I’m compelled to comment on
what can only be described as a diametrically opposed set of opinions
on the matter.

Here’s a perfect illustration taken from some comments on this blog entry
where I suggested that many CIO’s simply think that "awareness
initiatives are good for sexual harassment and copier training, not
security."

Firstly, here is someone who thinks that awareness training is a waste of time:

As to educating users, it’s one of the dumbest ideas in
security. As Marcus Ranum has famously pointed out, if it was going to
work…it would have worked by now. If you are relying on user
education as part of your strategy, you are doomed. See "The Six Dumbest Ideas in Security" for a fine explanation of this.

…and here is the counterpoint offered by another reader suggesting a different perspective:

Completely disagree. Of course you’re not going to get
through to everyone, but if you get through to maybe 80-90% then that’s
an awful lot of attacks you’ve prevented, with actually very little
effort. The reason I think it hasn’t worked yet is because people are
not doing it effectively, or that they’ll ‘get around to it’ once the
CEO has signed off all the important projects, the ones that mean the
IT Security team get to play with cool new toys.

What’s my take?

I think this is very much a case of setting the appropriate
expectations for what the deliverable and results should be from the
awareness training.  I think security awareness and education can bear substantial fruit.  Further, like the second reader, if the goals are
appropriately and realistically set, suggesting that 100% of the
trainees will yield 100% compliance is simply nonsense.

Again, we see that too often the "success" of a security initiative is
only evaluated on a binary scale of 0 or 100% which is simply stupid.
We all know and accept that we’ll never been 100% secure, so why would
we suggest that 100% of our employees will remember and act on 100% of
their awareness training?

What if I showed (and I have) that the number of tailgates through
access controlled access points went down over 30% since awareness training?
What if I showed that the number of phishing attempt reports to IT
Security increased 62% and click-throughs decreased by the same amount
since awareness training?  What if I showed that the number of reports
of lost/stolen company property decreased by 18% since awareness
training?  How about when all our developers were sent to SDLC training and our software deficiencies per line of code went down double digits?

What if I told you that I spent very little amounts of money and time
implementing this training and did it both interactively and through
group meetings and everyone was accountable and felt more empowered
because we linked the topics to the things that matter to THEM as well
as the company?

As to Marcus’ arguments
regarding the efficacy of education/awareness, he’s basically
suggesting that the reason awareness doesn’t work is (1) human
stupidity and (2) a failure of properly implementing technology that
should ultimately prevent #1 from even being an issue.   

I suggest that as #2 becomes less of an issue as people get smarter
about how they deploy technology (which is also an "awareness" problem)
and the technology gets better, then implementing training and education for issue #1 becomes the element that will
help reduce the residual gap.

To simply dismiss security awareness training as a waste of time is
short-sighted and I’ve yet to find anyone who relies solely upon
awareness training as their only strategy for securing their assets.
It’s one of many tools that can effectively be used to manage risk.

What’s your take?

Back in August before VMworld 2007, I wrote about the notion that given Cisco’s investment in VMware, we’d soon see the ability for third parties to substitute their own virtual switches for VMware’s. 

Further, I discussed the news that VMware began to circulate regarding their release of an API originally called Vsafe that promised to allow third party security and networking applications to interact with functions exposed by the Hypervisor. 

Both of these points really put some interesting wrinkles — both positive and potentially negative — in how virtualization (and not just VMware’s) will continue to evolve as the security and networking functions evolve along with it.

What a difference a consonant makes…

Several months later, they’ve added the consonant ‘m’ to the branding and VMsafe is born.  Sort of.  However, it’s become more than ‘just’ an API.  Let’s take a look at what I mean.

What exactly is VMsafe? Well, it’s a marketing and partnership platform, a business development lever, an architecture, a technology that provides an API and state of mind:

VMsafe is a new program that leverages the properties of VMware Infrastructure to protect machines in ways previously not possible with physical machines. VMsafe provides a new security architecture for virtualized environments and an application program interface (API)-sharing program to enable partners to develop security products for virtualized environments.

What it also provides is a way to make many existing older and consolidating security technologies relevant in the virtualized world given that today their value is suspect in the long term without it:

The VMsafe Security Architecture provides an open approach that gives security vendors the ability to leverage the inherent properties of virtualization in their security offerings.

Customers running their businesses on VMware Infrastructure will be assured that they are getting the best protection available – even better than what they have on physical infrastructure.

VMsafe adds a new layer of defense that complements existing physical security solutions such as network and host protection, shielding virtual machines from a variety of network, host and applications threats. This additional layer of protection can help enterprise organizations to dramatically increase the security posture of their IT environments.

There is then hope that a good deal of the visibility we lost in the limited exposure existing security solutions have across virtualized environments, we may get back.

Of course, this good news will be limited to those running VMware’s solutions and re-tooled versions of your favorite security vendor’s software.  What that means for other virtualization platforms is a little more suspect.  Since it requires the third party software to be re-written/adapted to use the VMsafe API, I can’t see many jumping on the wagon to support 3-4 VMM platforms.

Ack!  This is why we need a virtualization standard like OVF and a cross-platform signaling, telemetry and control plane definition!

So how does VMsafe work?

VMsafe enables third-party security products to gain the same visibility as the hypervisor into the operation of a virtual machine to identify and eliminate malware, such as viruses, trojans and key-loggers. For instance, security vendors can leverage VMsafe to detect and eliminate malware that is undetectable on physical machines. This advanced protection is achieved through fine-grained visibility into the virtual hardware resources of memory, CPU, disk and I/O systems of the virtual machine that can be used to monitor every aspect of the execution of the system and stop malware before it can execute on a machine to steal data.

…

VSAFE enables partners to build a virtualization-aware security solution in the form of a security virtual machine that can access, correlate and modify information based on the following virtual hardware:

• Memory and CPU: VMsafe provides introspection of guest VM memory pages and cpu states.
• Networking: Network packet-filtering for both in-hypervisor and within a Security VM.
• Process execution (guest handling): in-guest, in-process APIs that enable complete monitoring and control of process execution.
• Storage: Virtual machine disk files (VMDK) can be mounted, manipulated and modified as they persist on storage devices.

So where do we go from here?

With nothing more than a press release and some flashy demos, it’s a little early to opine on the extensibility of VMsafe, but I am encouraged by the fact that we will have some more tools in the arsenal, even if they are, in essence, re-branded versions of many that we already have.

However, engineering better isolation combined with brokered visibility and specific authorized/controlled access to the VMM is both a worthy endeavor that yields all sorts of opportunities, but given my original ramblings, makes me a bit nervous.

Alessandro from the virtualization.info blog gave us a description of the demo given at VMworld Europe that illustrates some of this new isolation and anti-malware capability:

A Windows XP virtual machine gets attacked with a malicious code that
copies away corporate documents but another virtual machine with
security engine is able to transparently recognize (a virtual memory
scan through VMsafe APis access) the threat and stop it before it
compromises the guest OS.

Steps in the right direction, for sure.  Since details regarding the full extent of anti-malware capabilities are still forthcoming, I will reserve judgment until I get to speak with Nand @ VMware to fully understand the scope of the capabilities.

I am sure we will see more claims surface soon suggesting with technology such as this will produce virtualized environments that are "more secure" than their non-virtualized counterparts.  The proof is in the pudding, as they say.  At this point, what we have is a very tantalizing recipe.

I’d suggest we’ll also see a fresh batch of rootkit discussions popping up soon…I’d really like to see the documentation surrounding the API.  I wonder how much it costs to sign up and be authorized to participate with VMsafe?

Some of the Determina functionality is for sure making its way into VMsafe and it will be really interesting to see who will be first first out of the gate to commercially offer solutions that will utilize VMsafe  once it’s available — and it’s a little unclear when that will be.

Given who demoed at VMworld Europe, I’d say it’s safe to bet that McAfee will be one of the first along with some of the more agile startups that might find it easier to include in their products.  The initial lineup of vendor support makes up some of the who’s-who in the security space — all except for, um, Cisco.  Also, where the heck is SourceFire?

When do the NAC and DLP vendors start knocking?

More detailed analysis soon.

/Hoff

The virtualization security (VirtSec) FUD meter is in overdrive this week…

Part I:
     So, I was at a conference a couple of weeks ago.  I sat in on a lot of talks.
     Some of them had demos.  What amazed me about these demos is that in
     many cases, in order for the attacks to work, it was disclosed that the
     attack target was configured by a monkey with all defaults enabled and no
     security controls in place.  "…of course, if you checked this one box, the
     exploit doesn’t work…" *gulp*

Part II:
     We’ve observed a lot of interesting PoC attack demonstrations such as those at shows being picked
     up by the press and covered in blogs and such.  Many of these stories simply ham it up for the
     sensational title.  Some of the artistic license and innacuracies are just plain recockulous. 
     That’s right.  There’s ridiculous, then there’s recockulous. 

Example:
     Here’s a by-line from an article which details the PoC attack/code that Jon Oberheide used to show
     how, if you don’t follow VMware’s (and the CIS benchmark) recommendations for securing your
     VMotion network, you might be susceptible to interception of traffic and bad things since — as
     VMware clearly states — VMotion traffic (and machine state) is sent in the clear.

     This was demonstrated at BlackHat DC and here’s how the article portrayed it:

Jon Oberheide, a researcher and PhD candidate at the
University of Michigan, is releasing a proof-of-concept tool called
Xensploit that lets an attacker take over the VM’s hypervisor and
applications, and grab sensitive data from the live VMs.

     Really?  Take over the hypervisor, eh?  Hmmmm.  That sounds super-serious!  Oh, the humanity!

However, here’s how the VMTN blog rationally describes the situation in a measured response that does it better than I could:

Recently a researcher published a proof-of-concept called
Xensploit which allows an attacker to view or manipulate a VM undergoing live
migration (i.e. VMware’s VMotion) from one server to
another. This was shown to work with
both VMware’s and Xen’s version of live migration. Although impressive, this work by no means
represents any new security risk in the datacenter. It should be emphasized this proof-of-concept
does NOT “take over the hypervisor” nor present
unencrypted traffic as a vulnerability needing patching, as some news
reports incorrectly assert. Rather, it a
reminder of how an already-compromised network, if left unchecked, could be
used to stage additional severe attacks in any environment, virtual or
physical. …

Encryption of all data-in-transit is certainly one well-understood mitigation
for man-in-the-middle attacks.  But the fact
that plenty of data flows unencrypted within the enterprise – indeed perhaps
the majority of data – suggests that there are other adequate mitigations. Unencrypted VMotion traffic is not a flaw,
but allowing VMotion to occur on a compromised network can be. So this is a good time to re-emphasize hardening best practices for VMware
Infrastructure and what benefit they serve in this scenario.

I’m going to give you one guess as to why this traffic is unencrypted…see if you can guess right in the comments.

Now, I will concede that this sort of thing represents a new risk in the datacenter if you happen to not pay attention to what you’re doing, but I think Jon’s PoC is a great example of substantiating why you should follow both common sense, security hardening recommendations and NOT BELIEVE EVERYTHING YOU READ.

If you don’t stand for something, you’ll fall for anything.

/Hoff

James McGovern over at the Enterprise Architect blog wrote a really fantastic Letterman’s Top 10 of mistakes that CIO’s make regarding enterprise security.  I’ve listed his in its entirety below and added a couple mineself… 😉

Use process as a substitute for competence: The answer to every problem is almost always methodology, so you must focus savagely on CMMi and ITIL while not understanding the fact that hackers attack software.

Ostritch Principle:
Since you were so busy aligning with the business which really means
that you are neither a real IT professional nor business professional,
you have spent much of your time perfecting memorization of cliche
phrases and nomenclature and hoping that the problem will go away if
you ignore it.

Putting network engineers in charge of security:
When will you learn that folks with a network background can’t possibly
make your enterprise secure. If a hacker attacks software and steals
data yet you respond with hardware, whom do you really think is going
to win the battle.

Over Rely on your vendors by relabelling them as partners:
You trust your software vendors and outsourcing firms so much that you
won’t even perform due diligence on their staff to understand whether
they have actually received one iota of training

Rely primarily on a firewall and antivirus:
Here is a revelation. Firewalls are not security devices, they are more
for network hygiene. Ever consider that a firewall can’t possibly stop
attacks related to cross site scripting, SQL injection and so on.
Network devices only protect the network and can’t do much nowadays to
protect applications.

Stepping in your own leadership: Authorize reactive, short-term fixes so problems re-emerge rapidly

Thinking that security is expensive while also thinking that CMMi isn’t: Why do you continue to fail to realize how much money their information and organizational reputations are worth.

The only thing you need is an insulting firm to provide you with a strategy:
Fail to deal with the operational aspects of security: make a few fixes
and then not allow the follow through necessary to ensure the problems
stay fixed

Getting it twisted to realize that Business / IT alignment is best accomplished by talking about Security and not SOA:
Failing to understand the relationship of information security to the
business problem — they understand physical security but do not see
the consequences of poor information security. Let’s be honest, your
SOA is all about integration as you aren’t smart enough to do anything
else.

Put people in roles and give them titles, but don’t actually train them: Assign untrained people to maintain security and provide neither the training nor the time to make it possible to do the job.

Here are some of my favorites that I’ve added.  I’ll work on adding the expanded explanations later:

1. Keep talking about threats and vulnerabilities and not about risk
2. Manage your security investments like throw-away CapEx cornflakes and not as a portfolio
3. Maintain that security is a technology issue
4. Awareness initiatives are good for sexual harassment and copier training, not security
5. Security is top secret, we can’t talk about what we do
6. All we need to do is invest just enough to be compliant, we don’t need to be secure
7. We can’t measure security effectiveness
8. Virtualization changes nothing in the security space.
9. We’ve built our three year security strategy and we’re aligned to the business
10. One audit a year from a trusted third party indicates our commitment to security

Got any more?

/Hoff

We had a little chat a few weeks ago at the apparent shock suffered by many a security professional in discovering that the three-legged stool of security was constructed of unequally leveraged legs of C, I and A.

Some reckon that by all practical accounts C, I and A should not be evaluated or assessed in a vacuum, but depending upon your line of business, your line of work and how you view the world, often this is how things get done — we have very siloed organizations, so it leads to siloed decision matrices.

Specifically, availability (or service delivery) in reality — despite what theory and purists espouse — often trumps "security" (the C and I functions.)  As distasteful as that sounds, this is endemic.  From operating systems focused on "usability" rather than security to routing protocols focused on rapid convergence and assumed trust as opposed to secure and authenticated mechanisms.

To wit (from the Renesys Blog):

Pakistan hijacks YouTube

Late in the (UTC) day on 24 February 2008, Pakistan Telecom (AS 17557)
began advertising a small part of YouTube’s (AS 36561) assigned
network. This story is almost as old as BGP. Old hands will recognize
this as, fundamentally, the same problem as the infamous AS 7007 from 1997, a more recent ConEd mistake of early 2006 and even TTNet’s Christmas Eve gift 2004.

Just before 18:48 UTC, Pakistan Telecom, in response to government order to block access to YouTube (see news item)
started advertising a route for 208.65.153.0/24 to its provider, PCCW
(AS 3491). For those unfamiliar with BGP, this is a more specific route
than the ones used by YouTube (208.65.152.0/22), and therefore most
routers would choose to send traffic to Pakistan Telecom for this slice
of YouTube’s network.                           
                              

Yes, this is really a demonstration of unavailability, but what I’m getting at here is that fundamentally, the core routing protocol we depend upon for the backbone Internet transport is roughly governed by the same rules that we depend upon whilst driving down a road separated by nothing more than painted lines…you simply hope/trust that nobody crosses the line and crashes into you head-on.

There is very little preventing someone from re-routing traffic.  This could result in either a denial of service (as the traffic would not reach its destination) or even something akin to an interception, "storage" and eventual forwarding for nefarious means.

So, here we have a case where again we depend upon a protocol that was designed to provide (A)vailability, yet C and I are left floundering in the wings.  We’ll no doubt see another round of folks who will try and evangelize the need for secure BGP — just like secure DNS, secure SMTP, secure…

This will hit deaf ears until we see the same thing happen again…

/Hoff

Read more…

There’s a little bit of serendipity floating about today and timing is everything.

Ed Skoudis (IntelGuardians) and I were chatting last week at ShmooCon regarding his previous research on VM guest escapes in hosted platforms and I raised a concern regarding my use of Parallel shared folders between my hosted XP installation and the underlying OSX host operating system.

I reckoned that this would be a very interesting vector for potential exploitation as it provides a direct pipeline to the underlying host OS and filesystem. 

While this bit of news isn’t about Parallels, it is about VMware’s comparable products (workstation, ACE, player, etc.) and it exploits the same vector.  From Computerworld:

February 24, 2008 (Computerworld)  A critical vulnerability in VMware Inc.’s virtualization software for Windows lets attackers escape the "guest" operating system and modify or add files to the underlying "host" OS, the company has acknowledged.

As of Sunday, there was no patch available for the flaw, which affects VMware’s Windows client virtualization programs, including Workstation, Player and ACE. The company’s virtual machine software for Windows servers, and for Mac- and Linux-based hosts, are not at risk.

The bug was reported by Core Security Technologies, makers of the penetration testing framework CORE IMPACT, said VMware in a security alert issued last Friday. "Exploitation of this vulnerability allows attackers to break out of an isolated Guest system to compromise the underlying Host system that controls it," claimed Core Security.

According to VMware, the bug is in the shared folder feature of its Windows client-based virtualization software. Shared folders lets users access certain files — typically documents and other application-generated files — from the host OS and any virtual machine on that physical system.

"On Windows hosts, if you have configured a VMware host-to-guest shared folder, it is possible for a program running in the guest to gain access to the host’s complete file system and create or modify executable files in sensitive locations," confirmed VMware.

There is currently no patch available.  The mitigation strategy is to disable shared folders.

It’s important to reiterate that this vulnerability does not affect VMware’s Type 1 (bare metal) virtualization platforms such as ESX.  However, on Friday, VMware released fixes for 5 vulnerabilities in ESX, some of which could be exploited to bypass security controls, gain access to data or result in denial of service.

/Hoff

{image from Anthony Martin Escapes}

UPDATE: Coverage of this is being hammed up quite a bit in the press to sound like it’s going to shake the very foundations of virtualization…not so much.  It’s an issue that is reasonably easy to address and represents what can be generally referred to as a relatively small attack surface.  Yes, it reinforces the need to think about VirtSec in the Type 2 (hosted) virtualization world, but as I said in the comments, it really depends upon how and why you’ve deployed client-side virtualization.

Right, so I’m in the UK from the 25th to the 27th. I’ll be tagging my usual suspects, but if you’re up for something in the evening, send me an email [choff @ packetfilter.com] or call my call router and it will find me:
+1.978.631.0302

/Hoff

You’ve no doubt seen the latest handywork of Ed Felten and his team from the Princeton Center for Information Technology Policy regarding cold boot attacks on encryption keys:

Abstract: Contrary to popular assumption, DRAMs used in
most modern computers retain their contents for seconds to minutes
after power is lost, even at operating temperatures and even if removed
from a motherboard. Although DRAMs become less reliable when they are
not refreshed, they are not immediately erased, and their contents
persist sufficiently for malicious (or forensic) acquisition of usable
full-system memory images. We show that this phenomenon limits the
ability of an operating system to protect cryptographic key material
from an attacker with physical access. We use cold reboots to mount
attacks on popular disk encryption systems — BitLocker, FileVault,
dm-crypt, and TrueCrypt — using no special devices or materials. We
experimentally characterize the extent and predictability of memory
remanence and report that remanence times can be increased dramatically
with simple techniques. We offer new algorithms for finding
cryptographic keys in memory images and for correcting errors caused by
bit decay. Though we discuss several strategies for partially
mitigating these risks, we know of no simple remedy that would
eliminate them.

Check out the video below (if you have scripting disabled, here’s the link.)  Fascinating and scary stuff.

Would a TPM implementation mitigate this if they keys weren’t stored (even temporarily) in RAM?

Given the surge lately toward full disk encryption products, I wonder how the market will react to this.  I am interested in both the broad industry impact and response from vendors.  I won’t be surprised if we see new products crop up in a matter of days advertising magical defenses against such attacks as well as vendors scrambling to do damage control.

This might be a bit of a reach, but equally as interesting to me are the potential implications upon DoD/Military crypto standards such as FIPS140.2 ( I believe the draft of 140.3 is circulating…)  In the case of certain products at specific security levels, it’s obvious based on the video that one wouldn’t necessarily need physical access to a crypto module (or RAM) in order to potentially attack it.

It’s always amazing to me when really smart people think of really creative, innovative and (in some cases) obvious ways of examining what we all take for granted.

I was browsing through the RSA 2008 conference agenda today and noticed that two of my talks and topics I blog about constantly were being featured as RSA keynotes!

How cool is that!?

It seems besides the talk I’m already giving, the fine folks @ RSA forgot to tell me that I was to deliver these, also.

They also accidentally attributed the speaking roles to someone else:

I’ll be busy sorting out this correction.

In the meantime, you can just preview them here:

Security and Disruptive Innovation

Information Centricity

😉

/Hoff

Last week I posted about a press release announcing a new product from Catbird called HypervisorShield.

I was having difficulty understanding some of the points raised in the press release/product brief, so I reached out to Michael Berman, Catbird’s CTO (also a blogger,) for a little clarification. 

Michael was kind enough to respond  to the points in my blog posting.

Rather than repost the entire blog entry, I have paraphrased the points Michael responded to and left his comments intact.  I think some of them invite further clarification and I’ll be following up with a Take5 interview shortly.  Some of the answers just beg for a little more digging…

Just to ground us all, here’s the skinny on HypervisorShield:

Catbird, provider of the only comprehensive security solution for virtual and physical networks, and developer of the V-Agent™ virtual appliance, today announced the launch of HypervisorShield™, the industry’s first dedicated comprehensive security solution specifically designed to guard against unauthorized hypervisor network access and attack.

Here are my points and Michael’s responses:

1. Hoff: The press release speaks to HypervisorShield’s ability to protect both the hypervisor and the "hypervisor management network" which I assume is actually referring to the the virtual interface of the management functions like VMware’s service console? Are we talking about protecting the service console or the network functions provided by the vKernel?

   Berman: We’ve built a monitor function that uses VMware APIs to watch for changes to/management of the virtual machines. We also have signature templates and customizable policies for network connections to the service console and the host.
    

2. Hoff: The press release makes it sound like protecting the hypervisor is accomplished via an IPS function that isolates VM’s from one another like Reflex and Blue Lane?

   Berman: With all due respect to our colleagues in this space, intrusion detection and protection is one element.  Catbird combines several technologies to extend separation of duties, dual control and strict change control to the virtual infrastructure. Deploying a signature for VMSA-2008-0001 is nice, but detection or prevention of a rogue virtual center administrator from pulling off a Societe Generale hack is priceless.
    

3. Hoff: What exactly does Catbird do (in partnering with IPS companies like SourceFire) that folks like Reflex and BlueLane? don’t already do.

   Berman: Rather than talk about the differences, let’s talk about the most important similarity.  I think I speak for all of us when I say that it’s like we are in a time warp to 1996 and I am explaining why you need a firewall for your DMZ.  Customers have little appreciation for the magnitude of the threats facing their virtual infrastructure.  Once we get past that, then we can talk about why Catbird is the best. (hint: we’re smarter, faster and stronger)
    

4. Hoff: How do you monitor the Hypervisor?

   Berman: We deploy a virtual machine that hooks into the vSwitch environment and that also monitors the ESX hypervisor via the VI API.
    

5. Hoff: You say in the press release that "hypervisor exploits have grown 35% in the last several years."  Which hypervisor exploits, exactly? You mean exploits against the big, fat, Linux-based service console from VMware? That’s not the hypervisor!

   Berman: I believe that the real threat to the virtual infrastructure comes from the collapse of separation of duties and the breakdown in implicit and explicit security controls within the virtual data center.  That being said, the hypervisor management application is probably the most significant area of the attack surface. If I can own the management GUI I own the hypervisor. If I can pull a stack smash against the ESX web server I own the hypervisor. If some poor shlemozzle configures Samba and NFS for the storage network then they become part of the attack surface too. You can blame us for some hyperbole, but the stat came from the CVE database. Gartner/451/Edison report that virtual infrastructure (VI) is less secure than physical and we have private data that shows people are deploying VI with no network security at all – this is just wrong.  I also think that writing about, or writing off the only risk as being some sort of red pill/blue pill hack is also wrong.

Thanks again to Michael for responding.  Look for a follow-on Take5 shortly to dig a little deeper.

/Hoff