September 8, 2018 at 1:04 pm
Sean Lange - Friday, September 7, 2018 2:39 PMOh goodness...I was just finishing up an application and ran into even more fun. This project is cleaning up sloppy code from a third party vendor. They call a number of webservices and store every single xml request and response in separate folders. This gets called hundreds of times a day and they had absolutely nothing to do any cleanup. Now instead of taking their approach which was to create a console application to just delete any file more than 2 days old and run it every night I decided that since the data really is somewhat important to store the data in the database (also theirs). After being told by them that the database can't store xml (sql server 2014) I told them I would write this myself. I created a new table to hold the xml and various other identifying information because we use this for analysis from time to time. I simply have to parse their horrific file names into something sensible and insert the data. All is fine except it took FOREVER to run the first time because they had accumulated over 400,000 xml files in a single folder. I just pushed to production earlier this week and it ran for hours and hours and hours. But the amount of daily load will be barely a few seconds to clean up.All was good until i poked around a little more and stumbled across a folder I hadn't noticed prior. There inside was one of the more shocking sights I have seen in a long time. There are 1.5 million files in that folder. Each a 2kb xml file which made up a total of about 6.5GB on disc. Keep in mind I already cleaned out three other folders at the level of insanity with the process I wrote. They put this in place less than 3 months ago. Who says third party software vendors are incompetent??? How does anybody consider this to be ok to just drop files in a folder like that with no intent whatsoever to go clean it up. At least it is Friday!!!
I've been going though similar stuff even with some in-house data/code. I'm working on a project that will simply drop 1.6TB of data from a 2.0TB ETL database. The data is severely duplicated in that all of the source data lives in files that they already keep forever. No one ever decided to have something that would cleanup the mess.
Then, I have quite the opposite from a 3rd party vendor where they simply drop all history whenever they feel like it instead of keeping the most recent data for troubleshooting.
The cool part about it is that it IS interesting work. For example, there are 23 tightly knit by FK tables with the offending data in them. Doing the initial drops is going to simply be to copy only the data I want to keep and then, after disabling all of the FKs, truncating the original tables and copying the data back (I didn't use the tables that I copied to for reasons that would take too long to explain here). The daily deletes that must happen after that have to be done in the right order to keep from having FK violations because it all has to be done while the tables are online and actively being used.
Lot's o' fun. Especially since I've never had the pleasure to whack 1.,6TB if data in a most prejudicial manner before.
--Jeff Moden
Change is inevitable... Change for the better is not.
September 8, 2018 at 4:38 pm
Grant Fritchey - Friday, September 7, 2018 8:50 AMHowever, I'm a notorious certifications critique who has made quite a few people quite angry with my rants against certs, so what do I know.
I don't think I'm particularly notorious for it, but I'm distinctly anti-cert (especially since the only one that was worth while has been abolished). I have no certs at all and I'm proud of it My pet hate amongst certs is any cert provided by a certain comms equipment manufacturer, because getting certified on seriously overpriced hardware and unmaintained software that that no-one in their right mind would ever buy is surely a clear indication of stupidity.
Tom
September 8, 2018 at 4:46 pm
jonathan.crawford - Friday, September 7, 2018 10:59 AMlet's be fair, everyone's SSN has been compromised for years now, so maybe we should stop considering it PII anyway...
I think you are confused. The fact that everyone's SSN is effectively public knowledge now makes it much more importantly PII than if it was hard to get hold of someon's SSN, not the opposite. If a batch of data about someone comes with an SSN, the SSN enables you to identfy who that batch is about so the presence of the PII in the batch guarantees that the batch is personally identifiable.
Tom
September 10, 2018 at 8:18 am
TomThomson - Saturday, September 8, 2018 4:46 PMI think you are confused. The fact that everyone's SSN is effectively public knowledge now makes it much more importantly PII than if it was hard to get hold of someon's SSN, not the opposite. If a batch of data about someone comes with an SSN, the SSN enables you to identfy who that batch is about so the presence of the PII in the batch guarantees that the batch is personally identifiable.
Disagree, because if the data is known to a malicious actor, then it's easier for them to spoof me than before, not harder. People should stop using it.
-------------------------------------------------------------------------------------------------------------------------------------
Please follow Best Practices For Posting On Forums to receive quicker and higher quality responses
September 10, 2018 at 9:06 am
Hello Threadizens,
Look, I could care less, but Hugo is quite chuffed about this, so I figured I'd share this with you. The 3rd Edition of Execution Plans is now available for download. Thanks to Redgate, it's still free. Go here, grab a copy. Also, feel free to share this with others. It'll make Hugo happy.
Also, just so we're clear, any issues with the book are on me. Hugo did his best to keep me on the straight path, but you all know how bad I can be. So send him praise. Send me the criticisms.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
September 10, 2018 at 9:14 am
Grant Fritchey - Monday, September 10, 2018 9:06 AMHello Threadizens,Look, I could care less, but Hugo is quite chuffed about this, so I figured I'd share this with you. The 3rd Edition of Execution Plans is now available for download. Thanks to Redgate, it's still free. Go here, grab a copy. Also, feel free to share this with others. It'll make Hugo happy.
Also, just so we're clear, any issues with the book are on me. Hugo did his best to keep me on the straight path, but you all know how bad I can be. So send him praise. Send me the criticisms.
Thanks Grant (and Hugo) really appreciate the new edition. Execution plans are one of my biggest downfall, so I'm certainly looking forward to trying another attempt of understanding and expanding the knowledge of them.
Thom~
Excuse my typos and sometimes awful grammar. My fingers work faster than my brain does.
Larnu.uk
September 10, 2018 at 9:53 am
Grant Fritchey - Monday, September 10, 2018 9:06 AMHello Threadizens,Look, I could care less, but Hugo is quite chuffed about this, so I figured I'd share this with you. The 3rd Edition of Execution Plans is now available for download. Thanks to Redgate, it's still free. Go here, grab a copy. Also, feel free to share this with others. It'll make Hugo happy.
Also, just so we're clear, any issues with the book are on me. Hugo did his best to keep me on the straight path, but you all know how bad I can be. So send him praise. Send me the criticisms.
Thanks for the new edition, Grant. I'll pay particular attention to the new material on Query Store. On a somewhat-related note, if anyone has a chance to see Erin Stellato's presentation on Query Store, I'd take advantage of the opportunity. It's a really cool presentation.
September 10, 2018 at 10:05 am
You could.
But you don't.
Seriously though, to everyone reading this. I don't care how much you already know about execution plans, go download and read this book. If you're new to the subject, it'll guide you right onto at least intermediate level. If you are experienced, you will still find bits and pieces that are new to you.
And if you see any mistakes, blame both Grant (for writing them) and me (for failing to catch them). And let us know. We love to learn too!
September 10, 2018 at 12:26 pm
Ebook and code samples downloaded. Word spread to the rest of our internal IT team.
WHOOHOO and thank you very much for the documentation.
September 11, 2018 at 1:34 am
Grant Fritchey - Monday, September 10, 2018 9:06 AMHello Threadizens,Look, I could care less, but Hugo is quite chuffed about this, so I figured I'd share this with you. The 3rd Edition of Execution Plans is now available for download. Thanks to Redgate, it's still free. Go here, grab a copy. Also, feel free to share this with others. It'll make Hugo happy.
Also, just so we're clear, any issues with the book are on me. Hugo did his best to keep me on the straight path, but you all know how bad I can be. So send him praise. Send me the criticisms.
Thank you, both!
September 11, 2018 at 3:54 am
If anyone is in the Carolinas or Virginia, my thoughts are with you. Also, I have a small lot in Florida you could camp out in if you need to evacuate and have an RV, camper or tents. No electricity or sewer unfortunately, but it's not far from restaurants and such. Just let me know.
September 11, 2018 at 11:12 am
Brandie Tarvin - Tuesday, September 11, 2018 3:54 AMIf anyone is in the Carolinas or Virginia, my thoughts are with you. Also, I have a small lot in Florida you could camp out in if you need to evacuate and have an RV, camper or tents. No electricity or sewer unfortunately, but it's not far from restaurants and such. Just let me know.
Same here, I have a couple of extra rooms at our place in Atlanta. I wish you the best.
September 11, 2018 at 11:45 am
My son and future daughter in law are heading out if there now. They moved there 2 months ago. Welcome to the neighborhood!!
Michael L John
If you assassinate a DBA, would you pull a trigger?
To properly post on a forum:
http://www.sqlservercentral.com/articles/61537/
September 12, 2018 at 5:24 am
OK, late summer / early fall colds are horrible...
Finally got some night-time cold drugs, popped them last night, slept like a rock, but now I'm having a heck of a time waking up this morning...
Ugh
September 12, 2018 at 6:32 am
jasona.work - Wednesday, September 12, 2018 5:24 AMOK, late summer / early fall colds are horrible...
Finally got some night-time cold drugs, popped them last night, slept like a rock, but now I'm having a heck of a time waking up this morning...Ugh
EEEWWWWWWWWWWWWWWWWW. Don't spread your germs all over the interwebs. I can't afford to get sick.
@=)
Feel better soon.
Viewing 15 posts - 62,176 through 62,190 (of 66,712 total)
You must be logged in to reply to this topic. Login to reply