Monday, September 19, 2005

A new solution becomes the "morning after pill" for Patch Tuesday

About two years ago as I was malcontently slumped over another batch of vulerabilities which required patches to remediate, it occured to me that even in light of good vulnerability management tools to prioritize vulnerabilities and patching efforts as well as tools to deploy them, the fact that I had to do either in a short period of time, well, stunk.

Patch too early without proper regression testing and business impact analysis and you can blow an asset sky high. Downtime resulting from "Patches Gone Wild" can result in more risk than potentially not patching at all depending upon whether the exploit is in the wild and the criticality of the vulnerability.

It was then that a VC contact turned me on to a company (who at the time was still in stealth mode at the time) - Blue Lane Technologies - who proposed a better way to patch.

Namely, instead of patching the servers reactively without testing, why not patch the "network" instead and apply the same countermeasures to the streams as the patches do to the servers?

Assuming all other things such as latency and resiliency are even, this would be an excellent foothold in the battle to actually patch less while still patching rationally! You would buy yourself the time to test and plan and THEN deploy the actual server patch on your own schedule.

Guess what? It works. Very well.

We started testing a couple of months ago and threw all sorts of nastiness at the solution. It sits in-line in front of servers (with NIC-card failover capabilities, thankyouverymuch) and works by applying ActiveFix patches to the network streams in real time. We took a test box behind the protected interface and had multiple VMWare instances of various Microsoft OS's and applications running. We hit it with all sorts of fun VA scanners, exploit tools and the like. Of course, the "boxes" were owned. We especially liked toying with MetaSploit since it allowed us to really play with payloads.

We "applied" the patches to the machine instances behind the PatchPoint Gateway. Zip. Nada. We couldn't exploit a damned thing. It was impressive.

"Ah," you say, "but any old NIPS/HIPS/AV/Firewall can do that!" Er, not so, Sparky. The notion here is that rather than simply dump an entire session, the actual active streams are "corrected" allowing good traffic to flow while preventing "bad" traffic from getting through -- on a per flow basis. It doesn't just send a RST and that $50M wire transfer to /dev/null, it actually allows legitimate business traffic to continue unimpeded.

The approach is alarmingly, well, so 20 years ago! Remember application proxy firewalls?

Well, if you think about how an FTP proxy works, one defines which "good" commands may be executed and anything else is merely ignored. A user could connect via FTP and type "delete" to his heart's content, but if "delete" was not allowed, the proxy would simply discard the requested command and never pass it on to the server. Your session was still up, you could continue to use FTP, you just could not "delete."

Makes sense, no?

If, for example, Microsoft's MS05-1,000,000 patch for, say, IIS was designed to remediate a buffer overflow vulnerability which truncated the POSTS to 1024 bytes, then that's exactly what Blue Lane's PatchPoint would do. If it *does* (on the odd chance) do something nasty to your application, you can simply "un-apply" the patch which takes about 10 seconds and you're in no worse shape than you were in the first place...

It's an excellent solution that further adds to reducing our risks associated with patching for a price point that is easily justified given both the soft-cost cost avoidance issues associated with patch deployment and the very real costs of potential downtime associated with patch installation failures.

Other manufacturers are rumored to be offering this virtual patching capability in their "deux ex machina" solutions, but I dare say that I have yet to see a more flexible, accurate and simple deployment than Blue Lane's. In fact, I've yet to see anyone promise to deliver the fixes in the timeframe that Blue Lane does.

I give it the "It Kicks Ass" Award of the week.

See Blue Lane Technologies for a far better explanation than this one.



Anonymous Dominic White said...

Could you possible explain a few things to me.

Why would you want to allow a malicious attack to send any traffic? I can't think how a legitemate $50 million wire-transfer would contain a malicious attack. It doesn't work like that, you either attack or you don't. In the rare case that a malicious user tries to sneak an attack in a legitemate request, and the request can still be salvaged by modifying the traffic, then some sort of intelligent failover should be used. I wouldn't want to give the attacker an additional opportunity and rely on the inline-patch having correctly prevented all traffic, especially if the source has already been labelled as malicious.

The only difference between this and an IPS is then that this modifies the traffic, instead of blocking it, allowing potentially malicious attackers some communication instead of none and possible allowing non-malicious users some access in very rare cases?

What am I missing?

8:02 AM  
Anonymous Dominic White said...

I actually sat down and did a bit of thinking about this. I wrote:

Mostly in response to your post.

8:56 AM  
Blogger Christofer Hoff said...


Thanks for the post. I apologize for not responding sooner. For some reason, I didn't see the comment field increment.

At any rate:

I think the fundamental departure between comparing the PatchPoint solution to IPS solutions of today illustrates the fundamental flaw of these IPS solutions and Information Security practices in general.

Today's products are based upon a block/pass mentality. The reality is that we ought to be focused on Information Survivability -- the ability to continue to operate and allow legitimate traffic or portions thereof to occur in the face of vulnerabile systems and active exploits meant to compromise them. I know far too many people who out of haste classified a vulnerability and resultant threat as critical, applied a manufacturer-supplied patch without regression testing and DoS'd themselves right out of business when their applications and or operating systems were affected in a way which precluded rollback except from backup.

In many cases this sort of practice -- especially in the face of a zero-day attack -- simply does not work.

Also, what happens when the vendor doesn't have a patch available? The Blue Lane engineers provide an ActiveFix in advance of the vendor supplied patch until the vendor-issued edition is issued. At this point, the manufacturer's patch is cycled in and the temporary one cycled out via the update process.

Or better yet, for those folks who actually apply blocking mode via in-line IPS systems, the resultant business reaction to being "protected" solidly while simultaneously taking the businss offline has been met with less than happy results. I've polled about 120 Security practitioners from the Fortune 2000 thus far and more than 80% of them don't turn on anything more than the primitive blocking rules of known bad that should never occur anyway. Yet they still plan on buying more IPS systems. Why?

Besides, single function IPS is being replaced by multifunction UTM solutions that provide far greater coverage in the spectrum
of the security landscape. Virtual patching will soon be a part of that landscape in addition to IPS solutions as an integrated solution.

What is attractive about the Blue Lane solution is the ability to make a business decision on how to balance the risk of patching impacts to the Enterprise quickly versus the ability to essentially buy time while still protected without the potential of halting business. Security "professionals" take an irrational position towards threat/vulnerability driven reactions rather than focusing on managing risk. Blue Lane allows folks to make a risk management decision rather than a purely driven technical decision.

It reduces OpEx from patching exercises and server recovery and/or the downtime associated by self denial of service. Is it the end of IPS? No. Never said it was. It certainly has a place in the toolbox of today.

The beauty of what Blue Lane offers is the ability to not touch the machines in question yet patch quickly at the network level to allow a more sane and rational regression testing process to occur. You are defending on your schedule, not the attackers.

To your other points:

In the case of virtual patching, the reality is that IPS systems are either (for the most part) signature driven based upon vulnerability or exploit irrespective of the context within which an "attack" is detected. Why would I apply a defense (virtual patching or otherwise) based upon an IIS server when the target machine is running Apache? Most IPS systems have no context about
the assets they protect. They apply blanket coverage across the entire landscape. It makes no sense. SourceFire and their RNA
is the one notable exception to this rule.

Once an "attack" is detected, the normal disposition for an IPS is to dump the entire session, send RST's, quarantine, etc. instead of filter out the offensive material and make the bad traffic good enough.

This is even more important as virtual patching is applied to internal users accessing resources/assets. If you're connecting to an internal resource (or more than one) and a singular event triggers your inability to transmit (even to a single destination,) does that make better sense than simply removing the harmful content and allowing access to resources not vulnerable to the "attack" in question?

The FTP illustration was made to compare the functional proxy-like capabilities of being able to remove harmful content, not to suggest that an IPS cannot provide this functionality -- of course they can.

In terms of the pairing of IPS and PatchPoint, they are built to do different things (currently) and I respectfully disagree with your opinion that they are "redundant." By using your argument, I should never have to patch a server because IPS systems of today are more than enough to protect resources from attack. That doesn't make sense.

As to Thomas Ptacek's second law, I find it hysterical that it's a law of MARKETING. I thought we were discussing reality:

"If your inline network security device claims to provide "virtual patching", the box must use the actual binary patch from [the vendor] to do it." (Actually, to be fair, he said "Microsoft" not "the vendor.")

If I had to wait for Microsoft to issue me an actual binary patch, I'd be out of business. That's also a slap in the face to most of the security researchers who disassemble the exploits to advise the vendors or themselves issue patches when the vendors can't or won't -- you know, just like the IPS vendors you speak of. I'll use your WMF vulnerability example...thank goodness for Ilfak Guilfanov and his WMF saved a lot of people's hides. Too bad it wasn't from Microsoft.

Blue Lane had an ActiveFix for that, too.

Thanks for the post. Glad you don't think it's snake oil, but it's not just a tarted up IPS.


3:21 PM  
Anonymous Dominic White said...

Hi Chris

Thanks for the response, but I think you missed my main point. I can quite happily conceed much of what you have said about the problems with IPS systems. However, my point was that virtual patching suffers from exactly the same problems, and more! To put it another way, virtual patching is just an IPS that tried to get something useful from a data stream instead of just dropping it. However, I pointed out why that was a bad idea (you don't want to let known bad traffic through). So virtual patching ends up as a bad idea on top of an IPS.

Similarly virtual patching can benefit from some of the IPS benefits, i.e. providing a temporary solution before a patch is released while regression testing occurs. Snort signatures are released fairly soon after the announcement of a vulnerability and the large user community ensures those signatures are honed to reduce false positives quite quickly. BlueLane has a much smaller community and user base, and would not be able to produce as high quality signatures in as short a time. Bear in mind an ActiveFix is just a hyped up term the marketing department use for 'signature'.

As for the claim that you can't tweak an IPS to be relevant for running services. I must disagree with you. Snort (for example) allows different preprocessors to be turned on/off and allows you to tune processors to certain products. Also specific ports can have specific processors attatched. Thus I could set up a snort instance watching for bad telnet traffic to port 78 only, if it was required.

So I don't see what benefits BlueLane could possibly be adding.

3:57 PM  
Blogger Christofer Hoff said...

...and I think you missed my main point (or perhaps I just sucked at making it.)

This is as much an operational business issue as it is technical one. In many cases, security teams (however operationalized they may be) don't deploy or manage patching. Server teams, helpdesks or IT Operations teams do. They don't know an IPS from a hole in the (network) (fire)wall.

This solution provides a more rational approach to applying security as a service layer to areas of the network that may not have IPS or even firewall controls in front of them. It's a classic keyhole approach that you assumed I was speaking about THE perimeter -- you know, the one facing the Internet.

Perimeters are multiplying while their diameters are collapsing. Internal segmentation and security service layers are being deployed en masse across the Enterprise and via service providers and mobile operators providing "clean pipes."

As such, as overlay models for securing distributed chunks of assets evolve, virtual patching allows the same teams responsible for patching to do so at a "choke point" in the network further up the stream while regression testing is done (by the server teams) to assess the impact a patch may have.

Again, since the server teams don't "do" IPS, they should be able to control the fate of how they patch.

Also, I am very familiar with Snort -- it's inline functionality included, but again, the assumption that IDS/IPS systems are technically capable of blocking signature-based attacks is not disputed. The manner in which they do so, is.

All data is not created equal, so applying policy driven IDP across elements of the network that may not even require security based on RISK doesn't make sense.

Why would I spend $100K+ on an IPS solution that isn't aware of the assets it is protecting and whether those assets are even vulnerable to attack.

SourceFire does (via RNA, 3D and Defense Center) and so does Blue Lane -- it doesn't even bother patching flows for an IIS exploit if it's heading toward an Apache web server.

It applies the right amount of threat/vulnerability/risk management by allowing good traffic through and making bad traffic good enough. There's so much less "tuning" needed with this approach.

As I mentioned, I already see IPS vendors touting virtual patching capabilities in their products. If you happen to own one of these, you get a double bang for the buck. Interestingly, the simple fact that IPS vendors who have every opportunity to simply block now see logic in the ability to strip potentially malicious content seems oddly in contrast to your point.

Are you saying they are just playing a marketing game, also? Perhaps, but there is merit to this approach.

By the way, not everyone uses Snort. Not every company has experts that can perform incident response or IPS tuning. Not everyone even turns on automated blocking. There's a reason for that.

Tell me the difference again between completely blocking a session that has been defined as 'bad' versus allowing data, once cleansed, to make it's way onward to its destination. Not everything is an 'attack.' In many cases, signature based detection yields huge false positives based upon a lack of context.

What if someone merely fat-fingered a submission to a form that wasn't actually malicious at all but matched the profile of an exploit. Your IPS solution would simply dump the entire session (perhaps like that $50M wire transaction I illustrated -- this really happened) rather than remove the offending data and send the cleansed traffic on its way, much like what would happen if the data made its way to a patched system NOT protected by an IPS.

Fundamentally, and most certainly as the focus changes from THE perimeter to the internal network, much more context must be driven to decide where, how, when and if defenses should be applied based upon the relative risk imposed by the threats and vulnerabilities associated with a specific asset.

Today's point solutions (even Blue Lane's) still mean that device sprinkling of stand-alone solutions occur, but virtualized UTM across service provider and Enterprises is picking up at an amazing rate. I'd be more than glad to talk to you about why I believe this given what I do for a living.

Again, I believe this to be an evolving illustration of the demarcation from Information Security to Information Survivability; focus on Risk Management and protect the assets that matter most.

4:52 PM  
Anonymous Anonymous said...

Blue Lane has already started saving money for IT departments, while IPS solutions were always increasing the cost of IT. I am in Charge of 2000 windows and 1300 unix servers and my department was always busy to deal with security patches(and patch tuesday was nightmare for us) prior to deploying Blue Lane solution. After Blue Lane not only we are able to reduce the cost rather we have been able to expand our IT services. Its amazing.

8:12 PM  
Anonymous Anonymous said...

Blue Lane VP Engineering interviewed for Podtech:

4:34 PM  
Anonymous Anonymous said...


10:18 AM  
Anonymous Anonymous said...

Getting all this much information on enterprise was interesting. Keeping this interest in mind, did we compile this informative article on enterprise.

5:24 AM  
Blogger singe said...

Hi Chris

Wow, it has been a long time since I looked at this post. It looks like you are conceding that this uses the same sort of traffic matching technology that an IPS would use. I agree with you that having a nicely pre-configured and managed service is of definite use to patching teams, but I think that is equally possibly with current IPS offerings. This doesn't invalidate Blue Lane as a technology however, in fact it is a positive.

My problem, is with the traffic 'cleaning'. As I mentioned in my blog entry on the subject, cleaning the bad parts out of known bad traffic is a bit like going to the trouble of finding out if patrons entering a bank are known criminals and then letting them in after removing their visible weapons. The traffic stripping adds another layer of potential vulnerability in that it could possible let through payloads for which Blue Lane hasn't released an active signature for yet, whereas a normal IPS would just block the whole connection.

As for your claim of dealing with false positives such as the wire transfer, you are being unfair in not conceeding that Blue Lane would also be vulnerable to false positives. In the case of a false positive, stripping the bad parts instead of blocking the connection isn't really an advantage, as you will be stripping actually good parts (due to it being a false positive). So in the case of a misconfigured IPS that blocks a wire transfer, the risk is that the transfer will be blocked which should initiate re-sending behaviour on any system designed to deal with amounts that large. However, in the case of Blue Lane, it would attempt to strip the bad part out of the wire transfer leading to all sorts of undefined behaviour, for example it could change the amount from $15 million to $15 because the extra 0'
s looked like a buffer overflow. This is far worse behaviour than just blocking the connection, and I'll go even further and say stripping bad traffic will be worse than blocking the connection in 'every' situation. This is the critical point that I believe *intrinsically* invalidates virtual patching techniques based on this IPS+strip method.

Thanks for taking the time to respond Chris. Hope you have a great 2007.

9:32 AM  
Blogger singe said...

P.S. singe == Dominic White (

9:33 AM  

Post a Comment

<< Home