Drinking the Kool-Aid
-
After a day at the MS BI conference, I'll say this. The stuff I've seen is very cool. And some people have had a lot of success.
You have to admit that Microsoft does a fantastic job of marketing their products. Sometimes it's overkill, but for the most part it's a constant, subtle push that MS products just work better, are cheaper, more customizable, something.
It's no different here and I have to admit that they've done a good job selling me here at the BI conference. I've done a little bit of a write up on the conference as well for today.
While I definitely see some great uses of BI and of the technology that Microsoft has built, I'm still not sure this is more than a niche. At least in the general, overall sense of the database market. It's complicated, and requires a lot of training, both the IT people building the solutions as well as the end-users. And it requires sponsorship. From the case studies and people I've talked to where it works well, they have buy -in throughout the organization when it's successful and they're mavericks working in their own world when it's less successful.
Overall I can see some of the BI stack, which needs a better definition by the way, being very useful. The SSRS portion and probably the Sharepoint technologies make sense and should get implemented pretty heavily in a large portion of installations. The OLAP/ETL/DW/Mining pieces seem like they'll still be less heavily used and probably less heavily understood, but I'm open to being proven wrong.
And I'd like to be. This stuff is cool.
-
Steve,
You seem to have been hovering around the difference between the database engine part of SQL and the BI/SSIS stuff that it getting tagged on and most experienced or senior DBA's will be struggling with the same issue.
A couple of years ago I worked on an Oracle regulatory reporting system for a large bank that had to crunch all of their data on a monthly basis. We were a hot team, and fundamentally understood the data, what it was about and how to get it in and out. Issues such as transactions and even backups were largely ignored – we needed to get data in and summarised fast – thereafter it was read only until the end of the next month. It was not what I would consider 'normal' database work.
A few months later Microsoft was hanging around our office and falling over themselves trying to get us on board as a (some precious metal) partner which we didn't do because we were Oracle branded – which in banks is a good thing. The discussions that Microsoft had centred around the fact that although their existing partners had good SQL skills they were more tuned to transactional systems than to reporting, BI or some other acronym and there was (and still is) a huge demand, and associated undersupply, of skilled organizations and people that can work with data that has already been transacted.
These days I don't work directly in the banks but people I know in the industry are commenting about how banks worldwide are struggling to implement Basel II. The main reason is that it is virtually impossible to understand all of the data in the bank, across various systems and do something useful with it within a politically dynamic organization. Even large vendors who say that they have a solution only scratch the surface. The problem that needs to be solved is far more complex than your average SQL expert will understand and you are right, it requires different skills, project management approaches, methodologies, operations and just about everything else.
The value proposition for banks and similar organizations that have to satisfy regulatory requirements is quite simple – if the projects are not implemented some regulator comes along, takes the licence away, closes the doors and switches off the lights. It is a business imperative build these systems, far more important than say interfacing with your suppliers' ordering system.
Although I don't deal in that market much anymore, there is a *huge* amount of work and opportunities in Basel II, SOX and other markets. These markets have a dependency on the BI, ETL, OLAP, etc. biased skills that you refer to. Companies like Microsoft, Oracle, SAP and IBM see huge money trees waiting to be picked and focus a lot of attention in drumming up skills and support in those markets. While the apparent evil doings of Microsoft, for example supporting dynamic languages or not is interesting and gaining mindshare, they see the big money in BI in large corporates – not a web user in Africa.
So, readers of SQLServerCentral who are considering BI related projects or work should give them serious consideration and not fear going up a financially blind or technically boring alley.
Simon
-
Cross posted from another thread, not necessarily in response to Simon. Though I do think Simon raises great points:
I think I've done a poor job of stating my BI position, so let me clarify a bit. I don't believe that BI will widely catch on because of the complexity and change required by many people and organizations. It's hard to implement, it's hard to get companies to buy in and retool their culture to deal with it and it's hard technology to understand.
I do think that BI, OLAP, mining, etc. can work great and do some amazing things when it's done well. I've seen it done well in some places and it can work great in the financial industries, in retail, etc. It's very, very cool stuff and it can be incredibly useful.
But it's hard to do right and it's been very expensive. So it's hard to widely deploy. 20-30 years ago I would have said the same thing of ERP systems. They were big, expensive, complex systems and weren't widely implemented. Sure the large organizations could do it, but most small and even medium companies wouldn't undertake it. SAPs vision early on of getting into all sizes of companies took a long time to develop.
I think BI is similar. I've heard for over a decade that it would be the next big thing. Even before XML. It's still a nichy technology, existing in pockets, and not widely implemented. It's definitely growing, but I'm just not sure it's growing at the rate that the press and vendors hype it.
One of the updates from the conference is that for the organizations and people that buy in, they love it. They want to do more. I was very interested to hear a few CTO/CIOs, including one from Energizer, talk that with SQL Server and the MS BI platform, the cost wasn't an issue. They easily see an ROI and it's definitely worth the cost. There was a caveat that people were expensive for the BI skills, but the software and tools weren't.
Michael Treacy talked about BI being almost a religon in some companies. I'll try to better write some thoughts on this over the weekend, but it seems to be true here. The Believers believe. But they caution it's a hard road and they're glad they're past it. I'm not sure how many people are willing to commit to the long road. At least in the US we're too next-quarter-ROI focused in too many places.
That's a generalization and without a doubt lots of you understand the OLAP and mining technologies and without a doubt there are more companies trying new projects. But I'm not sure if the new push by Microsoft and the lower cost will result in a great many more projects. I suspect it might bring about more failed projects than successful ones.
-
BI is one of those things that are sold well, because they have great sizzle: stop lights, cool blinking thingies, etc. However, it's never enunciated how much work it is to get those cool blinking thingies to work meaningfully, and how much it will cost.
The ultimate ETL tool is a SELECT statement. The ultimate BI tool is Excel. Just about everything else is a waste of money.
-
The data warehouse and BI concept started with Bill Inmon's OLAP concept and Ralph Kimball's enterprise data warehouse in the late 90. I was working for a software company at that time. In order to be ahead of the game, we started to build a data warehouse for our software. The data warehouse technology would probably take off but unfortunately the bubble bursted in 2000. A lot of technical company or .com company went down. Most of the companies started to save money by not upgrading or buying new software, laying off people and everything.
For the last eight years, I found out the technology going slower than the early 90. Most companies are very cautious about doing new things liked building a data warehouse. There are a lot of company selling ELT tools and BI tools.
In my last two companies that wanted to build a data warehouse, instead of spending money to buy those tools, the management decided SQL SELECT statement and DTS package can pretty much do the job.
There are lots of companies starting to think of building data warehouse for tracking, planning and forecasting the business. However they are moving very slowly because of the economy. Also there are not a lot of experts in those areas.
We wanted to hire a data warehouse developer, we got about 500 resumes, interviewed about 50 people and finally hired the guy who was working on the data warehouse at a hospital at the time we hired him. Now we found out the gem we hired turned out to be just a small stone. He was working on a small part of the whole project. His SQL skill is junior level, he said he was a SQL Server DBA, I did not know what kind of SQL Server he was working on, he did not even know how to read the log file and shrink the database. He said he installed the server and knew all kind of stuff but so far when the database went down, he could not even know how to start the server and figure out the problem. What a SQL Server DBA we have!!
Well for my opinion BI is not ready because of economy and technical skill problem. Microsoft tried to sell SQL Server 2005 as a BI ready database. However in my mind I also doubt how many company will take advantage of all the things that build in to do the BI work.
my 2 cents.
-
This sounds similar to an IT fad of the past, Decision Support Systems.
I remember being at a meeting with the company president where we were asked to introduce ourselves and what we do. There was a DSS guru at the meeting who was eager to share his vision with the president. When he started explaining his vision of why DSS was the wave of the future, the president stopped him. He explained that 5 years before he had been working on a Doctorate in this very subject, had come to the conclusion that it was completely worthless, and had dropped out of the program. He went on to explain that he had used a yellow note pad as his DSS tool when he made a decision to buy another company.
-
LOL! BI is indeed DSS, just dressed in different clothes.
The problems with DW and BI are emotional, not technical. Can't create indexes to solve those problems.
-
(The ultimate ETL tool is a SELECT statement. The ultimate BI tool is Excel. Just about everything else is a waste of money.)
A SELECT statement is algebraically flawed so it is a moot point it can do calculus, here is what I know you can do with ETL transferring DB2 AS400 deposits to SQL Server on the web in real time, I helped a previous employer make 3000 excel spreadsheets disappear no human intervention required through automation in less than one hour can a SELECT statement do that? I don't think so.
Relational algebra expert Chris Dates complains a SELECT returns a table.
Kind regards,
Gift Peddie -
(The OLAP/ETL/DW/Mining pieces seem like they'll still be less heavily used and probably less heavily understood, but I'm open to being proven wrong.)
Steve,
Data Mining is what changed Amazon to a growth company, every Pharmaceutical manufacturer in this country is required by law to use most of the Prediction modeling calculus and smart people they use SAS libraries with either SQL Server or Oracle, Banks use UDM to find the next borrower from previous loan data. The airlines need Time interval in calculus but they are not required like Pharma so they pretend there is no problem. The big data warehouse projects take time and money but the calculus is needed and used everyday, I know you can use Hyperion to decompose quarterly numbers.
Kind regards,
Gift Peddie -
"A SELECT statement is algebraically flawed so it is a mute point"
So? Who cares? It gets the job done.
"it can do calculus, here is what I know you can do with ETL transferring DB2 AS400 deposits to SQL Server on the web in real time, I helped a previous employer make 3000 excel spreadsheets disappear no human intervention required through automation in less than one hour can a SELECT statement do that? I don't think so."
I'm sorry, I'm having a real hard time understanding your point here.
"Relational algebra expert Chris Dates complains a SELECT returns a table."
So what? Obviously, people have used SQL with great success. I'm not wedded to relational algebra purity.
-
Again, it's very hard to understand what you wrote here, your sentences are too long. Are you coming from German?
I will respond to this: "every Pharmaceutical manufacturer in this country is required by law to use most of the Prediction modeling calculus and smart people they use SAS libraries with either SQL Server or Oracle"
SAS is the biggest waste of money yet. The pharmas just use Base and Macro, it's really just an OK but incredibly expensive calculator.
-
I have been a BI Professional at some large and small companies over the past 11 years. I have worked with many BI tools, primarily Cognos, over this period. My first point is that BI, in general, is very valuable to companies that understand how to use it and train their employees properly.
My second point is that I have been won over by Microsoft's BI offering. While it does have its bugs, it is clearly superior to Cognos and other products out there -- in terms of price point and capability. The caveat here is that it is a very comprehensive product and requires a significant time investment in training to understand it. I have self-trained myself in all of the components over the past 14 months. I fully understand the product but it takes time. I am in the process of replacing our Cognos products with a new SQL Server data warehouse and BI implementation at my company. After 3 months of implementation, the project has been very successful. We are doing things with these products that we could not do with Cognos and other standalone products. Management at my company has been very happy with what we have been able to produce -- with very little expense.
David
-
(The ultimate ETL tool is a SELECT statement.)
ETL(extraction transformation and loading) is calculus and I just gave you two things you cannot do with a SELECT statement.
Kind regards,
Gift Peddie -
Gift Peddie, could you explain the ETL is Calculus statement. I am genuinely interested.
-
Here is the link from one of the creators of relational calculus, and I also know Microsoft bought a company called Extera it make an aggregation engine for ETL sales was $500, mill before Microsoft bought it and I don't know how much Microsoft paid. But I know before SQL Server 2005 Informatica tool could cost 1 mill for Enterprise license. So enjoy the ETL matrix for Ralph Kimball.
Kind regards,
Gift Peddie
Viewing 15 posts - 1 through 15 (of 41 total)
You must be logged in to reply to this topic. Login to reply