Managing Your IT Data

Question

Managing Your IT Data

Steve Jones - SSC Editor

SSC Guru

Points: 736212
More actions
January 19, 2009 at 10:05 pm

#130078

Comments posted to this topic are about the item Managing Your IT Data

Viewing 10 posts - 1 through 9 (of 9 total)

You must be logged in to reply to this topic. Login to reply

Andy Leonard SSChampion Points: 10144 More actions · Answer 1

Enterprise metadata is important. Unfortunately, you learn that (or you will) when something goes wrong. I wrote about using SSIS to collect enterprise metadata at http://sqlblog.com/blogs/andy_leonard/archive/2008/08/28/ssis-design-pattern-collect-enterprise-sql-server-database-metadata-with-ssis.aspx. That article was prompted by an email from someone doing what I've needed to do before: detect changes in the source system schema before it broke the ETL.

On storing data in flat files: I've heard the argument that all data is stored in flat files.

An anecdote: My first database application used Access (and I'm admitting that here for all to see). But Access wasn't fast enough to lookup and return the data I needed. I needed screens to "pop." So I wrote my own data store using ini files. I created numeric indices using the keys of key/value pairs.

It flew.

:{> Andy

Andy Leonard, Chief Data Engineer, Enterprise Data & Analytics

agatti SSC Eights! Points: 836 More actions · Answer 2

We use LogClarity, by LogFidelity, to do the same thing. It gathers and stores raw log data, including SQL server traces, but also separately stores the data after it has been correlated and cleaned up. It has reporting and alerts and the data are stored in SQL Server. http://www.logfidelity.com/

SqlOnMyMind SSCertifiable Points: 5062 More actions · Answer 3

Andy Leonard (1/20/2009)
Enterprise metadata is important. Unfortunately, you learn that (or you will) when something goes wrong. I wrote about using SSIS to collect enterprise metadata at http://sqlblog.com/blogs/andy_leonard/archive/2008/08/28/ssis-design-pattern-collect-enterprise-sql-server-database-metadata-with-ssis.aspx. That article was prompted by an email from someone doing what I've needed to do before: detect changes in the source system schema before it broke the ETL.
:{> Andy

Andy, I don't know if it is just my computer, but a lot of the images in your article seem to be broken.

Dan Guzman - Not the MVP Hall of Fame Points: 3757 More actions · Answer 4

Dan Guzman - Not the MVP

Hall of Fame

Points: 3757

January 20, 2009 at 9:11 am

#929767

It's just you. Images are fine for me.

catwalker63 Valued Member Points: 56 More actions · Answer 5

Carla,

I'm not having any trouble with Andy's images either.

Steve Jones - SSC Editor SSC Guru Points: 736212 More actions · Answer 6

Steve Jones - SSC Editor

SSC Guru

Points: 736212

January 20, 2009 at 7:16 pm

#930032

Working fine in the UK.

Wayne Evans-440401 Ten Centuries Points: 1017 More actions · Answer 7

Andy

might it be some browsers can't handle multiple heights and widths?

from the bottom image:

Chrome, IE9, FF for windows handle it OK (UK too)

Wayne

Did you get access denied? Great the security works.

Ross McMicken SSCarpal Tunnel Points: 4436 More actions · Answer 8

I know some folks using Splunk for log analysis on enterprise internet connections that generate 150GB+ of log data per day. So far, they are pretty happy with the tool, but the learning curve to full usability is steep. A limited use version is available for free download, so it's pretty easy to play with.

j.a.c SSCrazy Points: 2283 More actions · Answer 9

Splunk is an excellent tool for mining tons of data (our systems log hundreds of Gb per day) and for correlating events across different devices/apps/log sources. While Splunk can make useful reports and dashboards it is not a BI platform. Now if only I can convince my superiors of this fact...