Data Overload

Question

Data Overload

Viewing 3 posts - 16 through 17 (of 17 total)

You must be logged in to reply to this topic. Login to reply

Gary Varga SSC Guru Points: 82166 More actions · Answer 1

David.Poole (8/10/2016)
...the chances of a decision returning to haunt you is low...

The use of this as the basis of decision making is, whilst pragmatic, the fundemental reason for modern commercial shorttermism IMHO.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!

ChrisM@Work SSC Guru Points: 186127 More actions · Answer 2

As part of an existing or anticipated PCIDSS [/url]requirement, we're looking at removing aged data relating to payment cards. For our purposes, no card equates to little further value to us.

We may be caught between a rock and a hard place - we have to retain transactional data for a minimum period of time and payment card data for a maximum period of time.

^{“Write the query the simplest way. If through testing it becomes clear that the performance is inadequate, consider alternative query forms.” - Gail Shaw}

For fast, accurate and documented assistance in answering your questions, please read this article.
Understanding and using APPLY, (I) and (II) Paul White
Hidden RBAR: Triangular Joins / The "Numbers" or "Tally" Table: What it is and how it replaces a loop Jeff Moden

Steve Jones - SSC Editor SSC Guru Points: 736270 More actions · Answer 3

David.Poole (8/10/2016)
I'm coming to the end of my career and looking back I can see huge amount of data collected, allegedly for auditing purposes, that was never accessed and no clear stakeholder for that audit data.
I see developers and DBAs trying to 2nd guess requirements that have never been stated or clarified. To be brutally honest the 2nd guessed requirements have two characteristics
Are only unnecessary in about 20% of cases
Are probably the clearest set of requirements you have to work with
One of the main reasons that outsourcing ends in tears is that the outsourcer can ONLY supply the requirements that are stated. They have little intrinsic knowledge of the particular business.
The 20% of cases where the 2nd guessed requirement is wrong is where the ever growing data comes from.
I think data retention policies are best argued from the legal perspective. I'd say operational systems be kept down to the minimum needed to fulfil their business function. Backend systems, I'd say, where there is no legal requirement, put a line in the sand and state 3 years retention on warm storage, 6 years on cold storage. Beyond 3 years the chances of senior management being in the same position is every diminishing so the chances of a decision returning to haunt you is low.
In 3 years time the powers that be will be in a position to blame a plethora of people who have left the business should an issue arise.

Great points.