Nixu Software Web Journal

In my previous blog entry on July 9, I went on record by saying that the DNS vulnerability announced on July 8 was somewhat theoretical, as there were no incident reports on the vulnerability being exploited in the wild. And while that was indeed the case at the time of writing, the situation changed for the worse rather drastically towards the end of July, as tools designed to exploit this vulnerability were made public. Not long after that, SANS reported the first confirmed instance of DNS cache poisoning utilizing this vulnerability (on July 30). Finally, to up the ante, Dan Kaminsky stated yesterday at Black Hat 2008 in Las Vegas that this security flaw might also make other systems and services such as email, FTP and RADIUS vulnerable.

As the best way for your organization to protect itself against these threats is to update your (recursive) DNS servers with the patches issued on July 8, please make sure that the appropriate software updates have been applied to your machines.

—advertisement—

Better yet, if you would like to make sure that your DNS servers are patched up automatically in cases such as this, please take a look at Nixu Secure Name Server: at only $495 per server per year, your DNS servers will be automatically patched up, and you will also have an access to our helpdesk in situations such as this. This is great value for money, as our customers have recently noticed.

—advertisement—

On a somewhat lighter note, Dan Kaminsky also had a very nice illustration aid with him at Black Hat 2008. Namely, a Finnish company called Clarified Networks had created a visualization of the pace at which DNS servers around the world were patched up since this vulnerability became into public domain. If you’re interested in checking it out, the illustration is available at YouTube.

As many of you have noticed, US-CERT has issued a new security advisory VU#800113 on July 8 2008 according to which all BIND versions are vulnerable to cache poisoning. This vulnerability only affects BIND servers in which recursion has been enabled. For further information on this vulnerability, please see US-CERT’s full advisory here.

Following, please find a summary of how this vulnerability affects Nixu Products:

Nixu NameSurfer Suite

The proprietary primary DNS server included in Nixu NameSurfer Suite IS NOT affected by this vulnerability. Secure64 DNS and/or NSD servers run as DNS secondaries to Nixu NameSurfer Suite ARE NOT affected. BIND servers run as DNS secondaries to Nixu NameSurfer Suite are affected ONLY if recursion has been enabled in them.

For users that have enabled recursion on BIND servers run as DNS secondaries to Nixu NameSurfer primary, we recommend that BIND servers are updated to the latest version.

Nixu SNS (Secure Name Server)

The BIND version included in Nixu SNS was affected by this vulnerability if recursion was enabled. To address this issue, all users running Nixu SNS in which automated software updates have been enabled, have received a patched version of BIND (9.2.4-28.0.1.el4) on July 9 2008 by 7am GMT/2am EST that addresses the vulnerability announced in VU#800113 advisory.

All in all, I think we have covered our bases rather nicely.

General Notes

More generally, although this vulnerability has attracted a lot of attention this time around, the DNS cache poisoning attack technique used to exploit this vulnerability has been known for quite some time. And interestingly enough, there have been no real-life reports of incidents in which this technique (providing false name resolution to recursive DNS servers by spoofing the address of an authoritative DNS server and guessing the right transaction ID) would have been used. Of course, it may have been difficult to spot that, but still…

As one of my tech savvy colleagues at Nixu stated (hello Juhani! ), perhaps the most interesting thing about this entire episode was the fact that Dan Bernstein suggested the security improvements included in the latest BIND releases (as well as certain other caching DNS products) already some years ago when this attack technique was first discovered. However, it took a bit of time and the involvement of Dan Kaminsky and Paul Vixie to make this vulnerability a mainstream news item - and before that, having most caching DNS server vendors reacting to this issue. Please visit Doxpara Research’s website for further details.

Of course, it’s always a big positive when there’s a push for people to update their DNS servers. After all, not everyone does that as often as they should. But software updates aside, I think there might also be another political agenda at play here. As ISC stated in their summary on this vulnerability:

“A weakness in the DNS protocol may enable the poisoning of caching recurive resolvers with spoofed data. DNSSEC is the only full solution. New versions of BIND provide increased resilience to the attack (emphasis is mine).”

Do they mean to say it’s high time for everyone to get their DNSSEC plans straight?

Over the last few months, there has been an increasing amount of coverage on cloud computing. Just last week, Infoworld run an interesting blog on the traction cloud computing has been gaining recently. Based on the developments outlined in this post, it seems like Red Hat is becoming increasingly serious on software appliances, virtualization and cloud computing - all areas in which we at Nixu Software have been active for good two years now. This of course is no wonder, as I’m sure the good people of Red Hat understand the great promise of these emerging technologies.

As far as DNS and DHCP services and geographically distributed enterprise networks are concerned, cloud computing and/or hosted virtual computing environments present these organizations with some rather interesting opportunities. Traditionally, to escape the complexities associated with running basic DNS/DHCP services in branch offices, there organizations have deployed hardware-based appliances on remote sites. However, now that there is an increasing number of service providers out there hosting virtual machines for a relatively low monthly fee, it actually might make sense to investigate the possibility of running DNS and DHCP services as virtual machines connected to the branch office over a VLAN (virtual LAN) network.

As far as I can see, this makes sense in two ways:

1. Reduction of maintenance and management overhead: as proper software appliances run on hosted virtual platforms can be managed over a secure, web-based user-interface and as their maintenance processes (sw upgrades) are automated, the DNS and DHCP servers running as virtual machines can be managed remotely from the central data centre location. Better yet, as the hosting service provider looks after the virtual computing environment against a low monthly fee, the corporate IT function no longer needs to look after the server platforms (which they have to do with traditional hardware-based appliances).
2. Reduction of operating costs: as DNS and DHCP services are not very CPU, memory or bandwidth-intensive applications, the cost of running these as hosted virtual machines is actually quite low. This is simply because most pricing models used by hosting service providers (as far as virtual computing environments is concerned) is the combination of CPU, memory and bandwidth usage. As DNS and DHCP servers consume very little of these scarce resources - and especially so in branch offices - the mothly cost of running a virtualized DNS and/or DHCP server can be as low as just $50 per month.

If you’d like to give this approach a go, there’s a two-step approach you can use to see what this would be like. The first step is to go to Nixu Software’s website and download our virtualization-ready software appliances for free 30-day evaluation (you can do the evaluation on VMware or Citrix Xen so you don’t need physical hardware for the evaluation). If you like what you see, you can then ask the hosting service provider of your choice to run our software appliances on their hosted virtual platform as “private” VMs that connect to your branch office(s) over VLAN.

As some of you may have already noticed, OECD (Organization for Economic Co-operation and Development) has recently started worrying about the utilization rate of the IPv4 address space. While the Internet penetration rate worldwide is only 15%, some 75% of IPv4 address space is already being used.

According to OECD, continuing with IPv4 is likely to create inequalities in the economic development going forward as only the elite (OECD refers to the rich western countries as “the elite”) would be able to enjoy the benefits and the new opportunites created by the Internet. As all of you guessed, the recommended solution to this problem is migration to IPv6.

For those of you who wish to dig deeper into OECD’s views on IPv6, please click here for a paper titled “Internet Address Space: Economic Considerations in the Management of IPv4 and in the Deployment of IPv6″. That’s a one hefty package for anyone interested in the policies and the macroeconomic considerations behind IPv6 migration.

After six weeks of blog silence, I’m back. Having made 107 posts here in the last 18 months, the writer’s block finally got me - there’s only so much one can say about DNS and virtualization, after all. From now on, I’m going to extend the scope of this blog from DNS and virtualization to also DHCP and IP address mgmt, as those two areas are also something we work with on daily basis. On this note, I’m planning on writing about Microsoft DHCP and UID & Network ID next week, so please stay tuned for more.

On a somewhat unrelated note, and as I’m sure most of you know, a blog really isn’t a blog unless the comment fields are activated. Also we tried doing that initially, but once we started receiving literally thousands of visits per day, the blog trolls came in in crowds. And so, after starting to receive hundreds of naughty links and comments on daily basis, we had to close them down in order not to become full time moderators.

Having said that, I realised during my six-week break that we should try to make this web journal (it’s really not a blog, is it?) more interactive. And on this note, I have a couple of questions to anyone who happens to be reading this:

1. Would You be interested in writing on DNS, DHCP, IP address management and virtualization?
2. Would You be interested in becoming a moderator for this blog / web journal? (assuming this wouldn’t be the one-pony show - yes, I’m the pony here - it has insofar been)

The reason I’m asking this is that assuming there’s enough interest, I think it might be quite an interesting experiment to convert this web journal into a BLOG (yes, with capital letters) where people who had something to say on DNS, DHCP and IP address management, could do so. And if someone was willing to moderate comments every now and then, all the better.

Nixu Software would be happy to continue to run this Wordpress platform for the purpose as well as to pick up all the server and bandwidth costs. And as I mentioned earlier on in the blog, we do get thousands of visitors each day here so there’s a good chance your blog entries would receive more attention than they would in most other places.

If you’re interested, please send us a note to info (at) nixusoftware.com. We’d appreciate a little background info (who you are and what you do) so that we don’t get fooled by the blog trolls! Ideally, I think it would be great to have 10+ writers so that no one would get hugely burdened - and of course to keep things interesting.

I’ll keep you posted on how this experiement takes off.

The more we talk with IT professionals, the more we find virtualized computing environments whose full potential has not been utilized. Although an increasing number of organizations have already committed to virtualization by investing in platforms such as VMware ESX Server and Citrix XenServer Enterprise Edition, these same entities still continue to run a number of applications and network services on traditional hardware platforms. From the business point of view, this doesn’t make much sense as the only way to fully utilize the virtualization investment is to consolidate as many services and applications on these platforms as possible.

This is especially true in computing environments where the traditional servers (or hardware-based computing appliances) are used to run services that are not particularly CPU intensive. For example, if you think about DNS or DHCP servers running on dedicated servers, these services consume a very low percentage of the available resources for 95%+ of the time. And bearing Moore’s Law in mind, the situation is not getting any better going forward. Yet at the same time, running dedicated servers is warranted from the information security point of view, as it makes these servers less compromised to software vulnerabilities. With several services running on single box, a vulnerability in any piece of software will compromise all services running on that server.

This is exactly why running DNS and DHCP servers as virtual machines makes perfect sense: it offers you the best of both worlds. From the information security point of view, a virtual machine (VM) dedicated to a specific service or application is inherently more secure than a general purpose server running several services on single O/S. Further, running dedicated DNS and/or DHCP servers as VMs also allows organizations to optimize the usage of the computing power at their disposal. This is simply because virtualization platforms do not waste CPU resources on dedicated DNS and/or DHCP appliances in the same way as traditional, hardware-based computing appliances do.

To find out where we’re coming from, please have a look at our White Paper on Virtualized DNS and IP Addressing Environments.

Since launching version 1.1 of Nixu DHCP Server in early February, we have been discussing always-on DHCP with dozens of organizations. To sum up these talks, we’ve found that although most entities running Microsoft AD in their network are generally happy with the availability of their DNS service, many of them have suffered from the loss of DHCP service at times. Quite typically, this happens when the operations are scattered around, as Microsoft DHCP servers do not support failover between a number of sites. So in practice, should you loose the Microsoft DHCP server for any reason - or the connection to the data center in which the DHCP server(s) are situated - the dynamic clients in your network will not be able to obtain an IP.

In these cases, we’ve found that the DHCP failover mechanism developed for Nixu DHCP Server 1.1 is ideally suited for network environments such as this, primarily for four reasons:

1. As we rely on network-based API in our DHCP failover mechanism, the two servers in the failover pair can be situated in separate network segments. While this may sound pretty logical at first, commonly used DHCP servers such as ISC DHCPD - or other DHCP products based on it - have actually required both servers in the failover pair to be located in the same network segment, which means that you end up having a pretty substantial number of servers if you want to ensure always-on DHCP in a bigger network. In our case, you can get away in most cases with just one DHCP failover pair.
2. Many of us are running Cisco or Juniper routers in our networks. The beauty of this fact is that both of them support two or more IPs in their DHCP relay configuration option. So in practice, if you run one Nixu DHCP Server in network segment x and another Nixu DHCP Server in network segment y, you can configure the IPs of these two servers in your routers. This makes sure that even if you lost the connection to DHCP server running in network segment x, you will still be able to obtain the IP from server in network segment y.
3. The third point would be ease of management and load-balancing. When running Nixu DHCP Servers as a failover pair, you can manage all configurations from the WebUI of the failover primary: all configurations are pushed automatically to the failover secondary over the network-based API. Further, under normal circumstances, the failover pair does software-based load-balancing which means that you are better equipped to handle sudden traffic peaks after e.g. long weekends when a large number of people return to work.
4. As the fourth point, if your organization is already running virtualization platforms (VMware, XenSource Enterprise Edition, etc.), you do not need hardware to run Nixu DHCP Servers. Just download the ISO installation media (software appliance) from our website and boot up a new vm. Compared to traditional computing appliances, this translates to lower costs and ease of platform maintenance.

Based on the feedback we have received, this approach seems to be the cure for most DHCP headaches. If you’re interested in trying out this approach yourself, you can do so by downloading Nixu DHCP Server for free evaluation from this link.

Some of you may remember my January 15 post on The Long Tail of Networking Software, which was an attempt to explain Chris Anderson’s Long Tail economics in the networking software context. To recap, the point I wanted to make was that the ability to distribute network appliances electronically over the Internet as software appliances (or virtual appliances) is likely to change the traditional distribution and pricing models used by the networking industry, and therefore the way in which networking software will be consumed in the future.

Interestingly enough, on February 25, Chris Anderson introduced his new thesis in Wired Magazine’s article Free! Why $0.00 Is the Future of Business. In this new piece, Chris argues that electronic goods and services that are consumed online and that have a marginal cost of zero, are all heading down the path to gratis. While this notion isn’t exactly controversial for people who have got used to consuming Google’s free tools and services, it’s still quite an intriguing idea. After all, we’ve all been taught that there’s no such thing as a free lunch. But could it be that we’ve all been taught wrong?

Well, I’m not sure about that, but I’m pleased to tell you all that today marks the launch of howismydns.com, an online service designed to validate and check the health of one’s public DNS servers. In addition to the DNS Zone Report tool, the website has also some other useful features such as WHOIS, forward and reverse lookups. The general design principle has been to focus on the essential tools an average DNS admin would find useful rather than including all the bells and whistles in the service. If you asked me, I’d say the DNS Zone Report alone is definitely worth a visit.

Best of all, howismydns.com is free. And by free, we mean absolutely free. There are no limitations as to how often or how many times you may use the service, nor does it require you to submit any information such as email address, phone number or other contact details that pesky sales reps would find handy. Just use it as you like, when you like, to have the health of your public DNS assessed for $0.00.

Last but not least, a word from the sponsor: this free lunch is brought to you by Nixu Software!

Last week marked the release of the new Microsoft Windows 2008 Server with experimental support for server virtualization in the shape of Hyper-V. As I have said here before, we have been convinced that Microsoft’s support will play a large role in server virtualization going mainstream. After all, by the time people have replaced their Windows 2003 Servers with 2008, they all can run software appliances on their Microsoft servers as virtual machines. Then again, this really is old news as Microsoft’s plan and schedule were known well in advance.

But although Microsoft admittedly dominated the media last week with their triple launch on February 27, it was actually VMware that ended up making the most interesting announcements, at least as far as server virtualization is concerned. To welcome Microsoft to the party, VMware had closed deals with all major x86 server manufacturers on factory-installed ESX Servers and announced those agreements at the VMworld Europe 2008 event held in Cannes last week:

• “VMware Announces Agreements to Embed VMware ESX 3i Hypervisor Across Broad Lines of Servers from Dell, Fujitsu-Siemens, HP and IBM”

• “Sun Microsystems and VMware Announce OEM Agreement to Add VMware Virtualization to Sun x64 Server and Storage Portfolio”

If this is not impressive, I don’t know what is! But perhaps even more importantly, this is a clear sign that server virtualization as an industry standard is no longer a question of if - it is a question of when. And this in turn portends a great market for software appliances in the not so distant future, as these two technologies are ideal complements that are likely to gain market share hand in hand.

And on this note, there’s one point I’d like to make to everyone looking at server virtualization: the only way to show ROI on the virtualization investment you make is to consolidate as many network services and applications on your virtualization platform as you possibly can. At Nixu Software, we’re convinced that D-services (DNS, DHCP) and IP address management is a good place to start this work, as these core network services are the lifeblood of virtualized network and computing enviroments. To download a white paper on the topic, please click here.

Some of you may be familiar with a presentation titled “Corrupted DNS Resolution Paths: The Rise of a Malicious Resolution Authority” held by David Dagon, Chris Lee and Wenke Lee from Georgia Institute of Technology and Niels Provos of Google, Inc at Network and IT Security Conference: NDSS 2008. To provide you with a quick summary, their research suggests that there are tens of thousands of rogue DNS servers on the Internet used for various kinds of malicious activities, leading to a rise of “second secret authority” that should scare the living daylights of all of us. For Associated Press’s coverage on the topic, please click here .

Now, it may be just me, but I find it rather interesting that while security professionals all over the world are worried by the “second secret authority” on the Internet revealed by this study, the mainstream media is simultaneously embracing a service called OpenDNS. According to respected publications such as Computerworld, The New York Times, and PC World, OpenDNS is a great “speedup tweak” for anyone who wants to have a quicker name to IP resolution than what one’s own ISP provides. OpenDNS claims that their service makes Internet more reliable, networks more secure, and provides insight into DNS activity. Better yet, the service is offered free of charge so what could possibly be wrong here?

Well, my answer to this would be: several things.

First, as everyone who knows their DNS can tell you, running open recursive DNS servers (i.e. DNS servers that allow anyone, anywhere to perform recursive queries) is one of the most basic DNS security mistakes one can make. It makes the DNS server more prone to cache poisonings and, perhaps even more importantly, can be exploited to amplify Denial of Service (DoS) attacks targeted at other DNS servers. Bearing this in mind, I find it somewhat ironic that OpenDNS advertises the IPs of their open recursive DNS servers (208.67.222.222 and 208.67.220.220) and improved network security on their website.

Second, OpenDNS openly admits that their service manipulates DNS data. Now, although I’m willing to give OpenDNS the benefit of the doubt and trust that their intentions are entirely benevolent, I think this is something that can be likened to the “second secret authority” on the Internet: no matter how good the intentions behind OpenDNS, the DNS data provided by the service has been manipulated in order to generate ad revenue. Paul Vixie has apparently labelled this approach as “typosquatting” and according to Wikipedia, OpenDNS has allegedly intercepted some requests for valid servers by landing a request for google.com on OpenDNS’s own page.

Third, although one can use OpenDNS to block name requests for sites that contain inappropriate content - for example schools using this service could block name resolution to adult entertainment and other websites unsuitable to minors - it does absolutely nothing to prevent an access to an inappropriate website should the user be smart enough to type in the IP address rather than the domain name of the site in his/her browser. Essentially, this service does not actually filter any content or block access to suspect websites as advertised (this should be done on web proxy / caching level) but simply relies on the assumption that people are not smart enough to use IPs as opposed to domain names. If they are, OpenDNS will not be able to do anything about it, as DNS is used only for name to IP resolution (and vice versa).

And so, although it probably is true that OpenDNS may provide a nice “speedup tweak” for someone situated relatively close to their recursive DNS servers, in most cases I would recommend against OpenDNS and sticking to the recursive DNS service provided by one’s own ISP, or to recursive DNS service maintained in-house. After all, using DNS for something it has not been designed for doesn’t come without downsides.