There was an article about databases getting bloated with tons of older data that's not needed. I've actually argued this point with various employers I've had and the need to archive and remove older data.
Usually without success.
The problems that I've seen is most applications aren't well suited to dealing with archived data. So if I remove a set of data, say everything older than 60 days, the data is gone. There's no option to search archived data, or even a note to ask someone to look for it. And if I just move this data to another database, I might get some performance benefits, but I'm not doing anything about the tremendous amount of data I'm creating or accumulating.
The article is based on a study from the Enterprise Strategy Group, and says that we compound this by having full-sized copies of production data on test and development systems. I think that's a good problem since I've had constant issues with groups that develop or test on subsets of data and not at full load. Especially with a self-tuning system like SQL Server where statistics and execution plans can swing dramatically if the data is different. To me that's a quote from someone needing to get their study more press or trying to sell a product.
I think we do keep too much data, but I also think that there's not enough of a cost for someone to dig in and determine how much can be pruned. It's the same problem with email; it's not always worth someone's time to trim their email (like salespeople), even when there's a cost to managing (administration, backup, etc) to keeping it around. Until the cost becomes high enough for people to handle this management task, I'm not sure things will change.
I'm not sure it's worth buying an expensive piece of software to do this, especially as you have to condition the knowledge workers to know or request archived data when it's needed. And then you need some way to move this data back online, because if it's always online, how have you solved the problem?
Steve Jones
PS: Vote for Service Pack 3 for SQL Server 2005. Right now there are no plans to release it. We need your vote so Microsoft will build it.
The Voice of the DBA Podcasts
The podcast feeds are now available at sqlservercentral.podshow.com to get better bandwidth and maybe a little more exposure :). Comments are definitely appreciated and wanted, and you can get feeds from there.
or now on iTunes!
- Windows Media Podcast - 26MB WMV
- iPod Video Podcast - 22MB MP4
- MP3 Audio Podcast - 4.5MB
Today's podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.
I really appreciate and value feedback on the podcasts. Let us know what you like, don't like, or even send in ideas for the show. If you'd like to comment, post something here. The boss will be sure to read it.