Problems In Building a Data Warehouse
-
Comments posted here are about the content posted at http://www.sqlservercentral.com/columnists/jchan/2832.asp
-
You got that right.
I've built a data warehouse (after a fashion). It's brilliant. I can predict stock costs for a year to within 1.8% of actuals.
Unfortunately the users don't really care.
As I see it the problem is largely the 80:20 rule. Top management are really only interested in the first 80% of the implementation. Once your sales orders are running through the system and you're controlling inventory OK (ish) they just don't care. So the onus for the last 20% is between IT and the business units themselves.
What I've found is that actually the people I am trying to work with to resolve data quality issues just don't care. They spend two hours a day fiddling with spreadsheets and take great pride in their accomplishments. These spreadsheets are used for top management reporting, and despite the fact that they aren't accurate another lesson I've learned is that top management DON'T ACTUALLY CARE because information at this level isn't reconciled to anything by anyone except me.
The amount of bull information flowing around our organisation is frightening, but as long as it is delivered people think they've done their job.
So when I give the user a report reconciling stock reports with sales (which show the flaws in our stock and shop floor reporting procedures) the user just ignores it.
The fact I can prove a 10% deviation between our production, sales, stock take and finance systems is irrelevant because nobody wants to know.
So, in reply to your article, here's what I learned :
1) Very few users can be trusted to tell you what their information requirements are because they see it as infringing on their job, which they perceive to be wasting hours creating bull spreadsheets.
2) It is extremely difficult for IT to resolve data problems themselves, if you try and take on this function (as I did) you end up doing everyone else's job for them, I actually did the stock take myself for a couple of months.
3) ANY manual spreadsheet of any size is AT LEAST 10% inaccurate, but is never checked. Manual errors tend to be small but cumulative.
4) The nature of computerised errors is different - they tend to be generally very accurate but miss the odd bit of info off. This is very easy for users to spot, in my case my reports are 95% accurate but as the users spot the errors quickly they don't trust them.
5) It may just be my users (they are pretty dumb) but they refuse to download a 95% correct report into a spreadsheet and correct it, they would rather create a manual spreadsheet every day. Despite the fact that it is inaccurate and takes 90 minutes longer.
I suspect they just like looking busy.
I give up. I am buying a goat farm in the Hebrides.
-
This article doesn't even begin to address what IT staff, the presumed readership of the article, can do to address the situation presented. Nothing here about determining measures, level of granularity, etc to help management and IT get on the same page. All we see is the problem. We all know that there's often a problem. What we need are recommended solutions and/or courses of action.
-
the title is Problems In Building a Data Warehouse.
-
Good article on some of the things DBA's should look out for when management approaches them. I think the main point here is that often, DBA's don't fully understand what data warehousing entails. As a result they often find themselves in this position of letting management set false expectations without being able to educate them as to the steps and participation required from the business side.
-
the most important word was "incrementally".
start small. simple procedure to keep a modest snowflake going (sales, custs, prods, time ... 4 tables). do NOT jump right into Analysis Serv. and all that.
then, start showing some users something. you will sometimes amaze them with just the basics. i find that a nice little ad hoc qry tool that they can pound on the snowflake with reaps great rewards. little by little the light bulbs start going on. then you can begin the great adventure
and about the 80/20 rule...it's about ROI. 20% of the effort gets 80% of the reward. so do some little things first. just to get users to open their eyes.
and i don't mean to belittle them. but they have so few tools at their disposal in most companies outside of the ERP.
-
Hello Janet Wong:
This is my first time to read your articles. However, after reading it, I begin to read all your previous publishings. They are full with passion from reality and I can not agree with any more....
Keep posting your great work!
thanks
David
-
Very good, to-the-point article. I'm working on a DW project now and it's good to be reminded of some of the issues we will be facing or already have faced.
One thing to point out is that a DW project is reverse of a normal project. Basically we need to deliver something first, then go back and figure out all of the requirements... very incremental process as Joseph pointed out. This is because the users will not know their requirements when you start... they may think they do but once you deliver some data for them to work with they will always change their minds.
David
-
Richard's reponse is kind of funny to me. My organization is the exact opposite. EVERYONE except IT is constantly worrying about data inaccuracy and trying to remove it. For anyone inside IT, unless the final revenue number is wrong, they are loath to change things. It's gotten so bad that our finance department has to build the data warehouse. Yes, I know it is crazy to have a full fledge developer in the IT department, but sometimes you need to do crazy things to get people's attention. (FYI, we are working on plans for moving the development into IT.)
Anyways, Richard, let me know if you are looking for a new job in the next month or two and are interested in taking on a significant role in a DW project. Our company needs IT people with your mindset!
Viewing 9 posts - 1 through 8 (of 8 total)
You must be logged in to reply to this topic. Login to reply