September 11, 2021 at 12:00 am
Comments posted to this topic are about the item The State of Data Growth
September 11, 2021 at 6:26 am
It is only a matter of time until every byte of every disk in the whole world is completely full of pictures of cute cats.
September 12, 2021 at 2:35 pm
Archiving data is a topic I don't think of often, until I read an article like this. In my previous job we'd write DELETE triggers that would write the deleted record to an archive table. You could, theoretically comprise the lifetime of some data that way, but I'd hate to have tried.
In my current job they do a better approach. If it's an app developed internally, then all records for data tables have two columns: CreateBy, CreatedOn, UpdatedBy and UpdatedOn. And they never delete data.
But archiving is still a challenge. Many times, we get close to running out of disk space. So far, that's been handled by opening a help ticket to have some additional storage added to a VM. But I wonder if it wouldn't be better to archive the data into some storage format, then remove it from the disk. We'd have the data; in case we'd ever need to view it. Generally, we only need the last 3 years of data, so holding onto 20 years' worth which is largely ignored, doesn't make much sense. At least to me. I've no idea how it could be done, but it's an interesting problem.
Kindest Regards, Rod Connect with me on LinkedIn.
September 13, 2021 at 12:23 am
The narrative isn't about Filestream and Filetable in SQL Server?
Just data growth?
Been writing for insurance practice management for 25+ years, every one of them will NOT see history touched in any way - ever. 5 - 6 tb per office isn't uncommon.
5 years ago i saw Filestream and Filetable as a solution to connecting db's to masses of document storage, but I found the toolset arcane and unusable to someone as remedial as myself. I have no problem storing it, just connecting to it in a verifiable way.
The storage size is not going away anytime in this industry.
September 14, 2021 at 8:53 am
"... we can see that a decade has brought us multiple orders of magnitude of data growth"
Not sure I would agree with that. Based on the figures quoted, we only created ~87.7 times as much data in 2020 as we did in 2011/2012, which would be less than one order of magnitude.
2EB per day for 365 days is 730EB per year in 2011/12, compared to 64,000EB in 2020.
Viewing 5 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply