July 7, 2005 at 4:52 pm
I saw this article on rethinking the RDBMS on C|Net and it caught my eye because, well, it had database in the title 🙂
At first I thought it might deal with object databases, or more intelligent databases that can handle more than character bits, perhaps decoding and indexing images, sounds, etc. That's an interesting idea and one that I thought should be adapted anyway to relational databases. If you could index Excel and Word documents along with your data, store them in their own datatype, it would be cool.
But this article talks about of all things, the flat file. Mostly with regards to newer processes that generate tremendous amounts of data, like billing records for things like cell phones, RFID tags, etc. Since much of this data is not updateable, it is kind of overkill to try and store it in an RDBMS. Indeed in the places where you are generating hundreds or thousands of GB per day, you'd be asking a lot of a DBA to keep a system up and running for this. A flat file does make some sense with an index applied, as suggested in the article.
It seems to me that this is a dBase or FoxPro solution, but that's just me.
However, there might be another way to deal with this, in fact, a way that would make this data usable on the fly. I saw an interview awhile back with Mike Stonebraker of Streambase that talked about their streaming database technology. They have a way of dealing with huge amounts of data in memory as a "stream" that passes through and then can be stored in any number of ways, whether that be flat file or RDBMS. If you don't know anything about this technology, take a few minutes and read about it. It's pretty cool.
I don't usually generate hundreds of GB per day of data, but there are a few times that I have dealt with that volume of data for a short period of time, usually in running profiler on a heavily loaded system. Being able to have something do some stream processing on that data, maybe even culling it in real time for performance problems or patterns of activity, would be a neat addition to SQL Server.
Probably not for this edition, but for SQL Server 2008. Maybe the tools group will license this technology. Bug Euan Garden at PASS (and register with the "SSC" source code).
Steve Jones
July 11, 2005 at 8:00 am
This was removed by the editor as SPAM
July 11, 2005 at 1:51 pm
I don't know if this is relevant but in C# leaving flat file in memory for a long time comes out of the box, that is what was used to create the Caching Application block. One of the child classes of TEXTWRITER abstract class Memory stream is can do that and more by default. That is the reason if you want to manipulate images you have to know Memory stream just mark some properties read only. Hope this helps.
Kind regards,
Gift Peddie
Kind regards,
Gift Peddie
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply