I thought this ComputerWorld article was a good look at one potential way to manage all that data that we consider archiving. The article talks about identifying that data that can be thrown away, not just the data that needs to be retained.
I think this is harder to do in databases than with email and much other data. Often we can bucket out data from other systems and easily determine that this data is 5 years old and can be removed. Doing that with database data, often which still retains some value over time and is important for ensuring correct summaries, is harder.
In most of the applications I've worked with, the database is designed so that summaries are prepared in real time, using all the data available. While it might be possible to summarize old data and create "summary records", as a DBA, I have an inherent concern over doing this. What if we want to change the way we summarize data? What if want to mine data for "what-if" scenarios? Being able to look back at the details might be important.
I do think that we need to pay attention to storage costs a bit more. Even with relatively cheap disks, as we grow larger and larger storage systems, backup/recovery, and management start to play a larger role and could be helped by less data.
I don't know how easy it even is to create these classifications and manage the data removal, but I'm sure that we'll see some products that promise to help us soon.
Steve Jones
Steve's Pick of the Week
'Hello, I'm a PC' - The new Microsoft commercials are now available. I didn't think the Seinfeld/Gates ads were very good, though it was humorous to see Bill Gates make fun of himself. These are more interesting, but I'd love if they'd focus for 5 or 10 seconds on one person's work.
The Voice of the DBA Podcasts
The podcast feeds are now available at sqlservercentral.podshow.com to get better bandwidth and maybe a little more exposure :). Comments are definitely appreciated and wanted, and you can get feeds from there.
or now on iTunes!
- Windows Media Podcast - 30.5MB WMV
- iPod Video Podcast - 23.6MB MP4
- MP3 Audio Podcast - 4.8MB
Today's podcast features music by Incompetech. Kevin Macleod has some great compositions in all genres of music. Check him out at www.incompetech.com.
I really appreciate and value feedback on the podcasts. Let us know what you like, don't like, or even send in ideas for the show. If you'd like to comment, post something here. The boss will be sure to read it.