Data Overload

• Gary Varga

  SSC Guru

  Points: 82166

  More actions

  August 10, 2016 at 4:31 am

  #1894736

  David.Poole (8/10/2016)

  ---

  ...the chances of a decision returning to haunt you is low...

  The use of this as the basis of decision making is, whilst pragmatic, the fundemental reason for modern commercial shorttermism IMHO.

  Gaz

  -- Stop your grinnin' and drop your linen...they're everywhere!!!

• ChrisM@Work

  SSC Guru

  Points: 186127

  More actions

  August 10, 2016 at 6:50 am

  #1894759

  As part of an existing or anticipated PCIDSS [/url]requirement, we're looking at removing aged data relating to payment cards. For our purposes, no card equates to little further value to us.

  We may be caught between a rock and a hard place - we have to retain transactional data for a minimum period of time and payment card data for a maximum period of time.

  ^{“Write the query the simplest way. If through testing it becomes clear that the performance is inadequate, consider alternative query forms.” - Gail Shaw}

  For fast, accurate and documented assistance in answering your questions, please read this article.
  Understanding and using APPLY, (I) and (II) Paul White
  Hidden RBAR: Triangular Joins / The "Numbers" or "Tally" Table: What it is and how it replaces a loop Jeff Moden

• Steve Jones - SSC Editor

  SSC Guru

  Points: 728164

  More actions

  August 10, 2016 at 10:36 am

  #1894854

  David.Poole (8/10/2016)

  ---

  I'm coming to the end of my career and looking back I can see huge amount of data collected, allegedly for auditing purposes, that was never accessed and no clear stakeholder for that audit data.

  I see developers and DBAs trying to 2nd guess requirements that have never been stated or clarified. To be brutally honest the 2nd guessed requirements have two characteristics

  • Are only unnecessary in about 20% of cases
  • Are probably the clearest set of requirements you have to work with

  One of the main reasons that outsourcing ends in tears is that the outsourcer can ONLY supply the requirements that are stated. They have little intrinsic knowledge of the particular business.

  The 20% of cases where the 2nd guessed requirement is wrong is where the ever growing data comes from.

  I think data retention policies are best argued from the legal perspective. I'd say operational systems be kept down to the minimum needed to fulfil their business function. Backend systems, I'd say, where there is no legal requirement, put a line in the sand and state 3 years retention on warm storage, 6 years on cold storage. Beyond 3 years the chances of senior management being in the same position is every diminishing so the chances of a decision returning to haunt you is low.

  In 3 years time the powers that be will be in a position to blame a plethora of people who have left the business should an issue arise.

  Great points.

  Follow me on Twitter: http://www.twitter.com/way0utwest
  Forum Etiquette: How to post data/code on a forum to get the best help
  My Blog: www.voiceofthedba.com