What do people buy most of before a hurricane? Wal-Mart determined that batteries are the number ones sales item before a hurricane, but the second most sold item was surprising: pop-tarts. That's the type of intelligence that can come from analysts asking the right questions about their customers and then IT delivering systems that can help them get an answer.

More and more companies are starting to deal with big data. Big data is usually seen as extremely large sets of data. Those data sets  often exceed the capability of what a single database server can handle, and place a strain on existing IT infrastructures, especially when there is an explosion of new datab types. Hadoop is an open source framework that's designed to deal with large sets of data and help with the processing and analysis of these large data sets.

Microsoft recently added support for Hadoop to SQL Sever. It's available for the Parallel Data Warehouse, which is a highly specialized version of SQL Server integrated with specific hardware. However I am sure this will eventually make its way to other editions over time.

Update: I made a mistake, there is a connector for non PDW SQL Server 2008 R2 instances.

I don't know if I would recommend learning Hadoop, but the techniques of processing large sets, and performning analysis on big data is important., Even more important might be the writing of well performing T-SQL code that will be used. I'd recommend you read about ways to write better code, and learn to integrate those skills into your daily work. You don't always need them, but you often don't know that when you are writing the code.

It doesn't take much more effort to write better code the first time, but it does take more effort to learn to do so. It's an investment in your career that you ought to be regularly working on to become one of those talented people that is in demand in the future.

Steve Jones

The Voice of the DBA Podcasts

• Windows Media Podcast - 19.5MB WMV
• iPod Video Podcast - 14.9MB MP4
• MP3 Audio Podcast - 3.5MB MP3

The podcast feeds are available at sqlservercentral.mevio.com. Comments are definitely appreciated and wanted, and you can get feeds from there. Overall RSS Feed: or now on iTunes!

Today's podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

You can also follow Steve Jones on Twitter: