Streams
-
I wrote about Streambase some time ago when an interview with Michale Stonebreaker caught my eye. The idea of working with a stream of data for analysis instead of a traditional RDBMS is an interesting one and it certainly works better in some high volume situations. Now it appears IBM is releasing a product specifically to deal with large streams of data. It looks like a slightly different project than what Streambase has, but it would be interesting for projects like image or text recognition. I know we've gotten better, but there's definitely more work to be done here.
The whole idea of processing a stream of data and working with it, when you have incomplete information, is fascinating to me. How do you make decisions based on data that's flowing by and rapidly changing? I know it's a tough problem and one that I worked on years ago. We struggled with pricing data from the stock markets, which received in a stream from a satellite link and couldn't keep up with on our SQL Server 6.5 system and VB based inserts.
Streambase has come up with "StreamSQL", which is an extension of the SQL language designed to deal with streams. If you register with them, you can get some interesting information and download a trial. Since I don't have a good stream of data, I didn't bother, but I did think some of the white papers were interesting.
The pace of data acquisition and the variety of types of data has grown tremendously over the last decade I've been working with SQL Server and the next version, SQL Server 2008, really seems geared to help work with diverse types of data. And we will all need skills to work with larger, and more continuous streams of data as the world evolves into one with more and more digital information.
I'm not sure where SQL Server should go after 2008, but I do know that there are two areas it could use work in: large streams of data and parellel processing. Maybe we'll see one of these in 2011.
-
Hi Steve,
Interesting topic, indeed.
To me streaming databases will become very succesfull, especially in the domain of robotics.
If there is 1 area where online processing of mass data streams will be necessary, then it is the area of robotics.
Robots will process large amounts of data from the surrounding reality, interprete the incoming data against their current model of the world, decide for further action or learn from their situations and therefore adjust or augment their current model of the world. The robot's current model can be representated in a database, also its toolbox full with decision logic's .. The incoming streaming data will be managed by a streaming database.
With a database at hand in a large memory, robot programming will be much easier.
So, up to 'SQL Server streaming mobile' ..
Regards,
Franky
Franky L.
-
Interesting timing on this editorial, Steve.
I'm currently researching StreamBase and some others for potential use in some projects at my company. I saw some of the marketing flurry a couple of years ago when Stonebraker first started promoting his product, but it was only of passing interest at that time as there wasn't any applicability to what I was doing (and because Stonebraker's marketing fuzz was a bit offputting).
If history is any indication, I think you're right that we'll see MS try and tack on some form of stream processing engine. They're far more likely to do that than to actually address the fundamental flaws with their product that make such kludges necessary.
Hope your trip is going well.
Take care,
TroyK
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply