Start Fixing Your DB with Better Code
- patrickmcginnis59 10839 - Friday, November 24, 2017 10:42 AM
Whose definition of "programming correctness"? Yours? You're very likely correct there as we both know that we disagree with each other in that area on many fronts. That's what makes this community great... the innovation that can occur through disagreement. If it's not your programming correctness, then whose? Supposed "Best Practices"?
If by "programming correctness", you mean "accuracy", then you need to provide examples for to substantiate your rather insulting comment.
--Jeff Moden
RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.Change is inevitable... Change for the better is not.
Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally) - Jeff Moden - Saturday, November 25, 2017 11:32 AM
correctness is paramount, and its pretty much where all the big concerns are nowadays, heck for that matter I know plenty of historically important papers were equally concerned. even more so, I see more and more often the voice from the tech rank and file that this is a growing conflict between tech and management about this sort of thing.
about the insult part, I'm sorry if you feel insulted. I should have said programming correctness isn't "as important" to you, not categorically unimportant and that's my mistake in wording.
- Jeff Moden - Saturday, November 25, 2017 11:32 AM
well looks like the text that I have quoted has changed but you do bring up new points with this edition. I do think disagreement makes the community great.
You do now show some "out of touch-ed-ness" by pretending that programming correctness is only my definition, something I made up, so I now think that my accidental typing is now sort of serendipidous (lol spelling). Whereas I had originally congratulated you (and of course your department) because programming correctness wasn't as great a priority as it is with me (and its true, I typed it wasn't important rather than it wasn't as important as I should have), now you are labeling it as "my invented term" I guess, you know, "pats definition", and with that I guess we are so far apart on this that I'll just have to let that be, there aren't enough keystrokes in the universe to help me on this one honestly, so with that I think I'll just be satisfied with being the big old meanie here.
Its also true that I thought you were lucky, and maybe that belittled your teams hard work, but heck, being successful is always a sign of good fortune in my eyes, maybe some folks think that hard work always brings success and maybe I'm not so convinced on that one but I did not mean to belittle your teams hard work. It was my mistake, given that I should by now know how eagerly posters here seek insults in posts that might not actually have them.
Its true that I don't like your disdain for "best practices", its a terrible thing that we cannot make a study and a determination of the best way of doing things. Heck maybe in your eyes I'm the bad guy by not knowing what you mean, and in reality maybe I'm just disdainful that you point to anything called best practices that aren't even good practices as a reason why best practices should be invalided? Heck if something is labeled best practices but in actuality aren't even good practices, does this have to invalidate what best practices are intended to be? I've never gotten that idea that there is something inherently evil in publishing guidance using expertise, logic and indeed reputation. Don't we all ask everyone to be critical of what we read on the web, and can't that very ability to judge the source, reputation and logic help us make "best practices' actually useful? "Best practices" means that someone has figured out the best way to do something and by golly its my very wish that I can go to people smarter than me. Yes, I judge them by reputation and by the logic in what they say, and I'm happy that people are sharing their knowledge and feel blessed that I'm able to separate the wheat from the chaff and I do feel some pity and embarrassment for you that you cannot.
Still, the stupidest post from you is that "two programming languages is a tower of babel" thing, but the "best practices" idiocy isn't too far behind honestly. This current stream of consciousness isn't really all that bad in comparison.
edit: I completely forgot about that EBay mess, that one actually upset me, if we could find time to chat about that one, it would be awesome!@
- patrickmcginnis59 10839 - Saturday, November 25, 2017 1:18 PM
You're incorrect with either notion but you are entitled to your opinion. When I comes to T-SQL, I'm very concerned about "programming correctness". I'm also concerned about "programming correctness" on the front-end because most of my job has been to track down problems caused by "incorrectness" that causes both inaccuracy and performance problems.
--Jeff Moden
RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.Change is inevitable... Change for the better is not.
Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally) - patrickmcginnis59 10839 - Saturday, November 25, 2017 2:30 PM
To be sure, it's not "Best Practices" that I disdain. In fact, I embrace proper "Best Practices". It's what people call "Best Practices" that I disdain. It seems like if enough people say something often enough, it becomes a "Best Practice" even when it's not. A classic example of that is the ignorance that surrounds the "Best Practice" of not doing direct date math. Other favorites are when people in high position insist that all code must be portable , that nothing proprietary to a given RDBMS engine should ever be used, and that dynamic SQL should never be used.
As for ...
I've never gotten that idea that there is something inherently evil in publishing guidance using expertise, logic and indeed reputation.
That's a huge part of the "Best Practices" problem I'm talking about. I know of a particular very well known author that supposedly proved that an XML splitter is faster than using the likes of the DelimitedSplitN4k function. He actually wrote a full up "holy grail" article complete with test data generation code and a test harness and then also wrote it into a book in a full "Alice's Restaurant" fashion. When I tried to tell the author that he'd made several serious mistakes (first, he changed the code rather than testing it as it was written) and tried to explain the biggest one, which was his extremely poorly written test data generator, he told me he wasn't going to restest and shut down the ability for anyone to post any further comments on the article. He has "expertise, logic, and indeed reputation" but it turns out that he's completely incompetent and unaware when it comes to that kind of test and indeed an idiot for not looking into the other possibilities that others were trying to tell him. I fear for his followers because they now think that using the XML splitter is a "Best Practice".Another fine example is from a moroff on LinkedIn that claimed that using rCTEs was a best practice for generating numeric sequences. Even after I demonstrated that a well written While loop would beat his rCTE example and 3 other methods that blew the doors off both, he cut me off, deleted my posts, and maintained the he was right and that it was a "Best Practice". Yeah... thanks for sharing. The really sad part is that his customers think he's some sort of a genius for writing such "clever" code.
On the "tower of babel" thing... no... it wasn't stupid. As you say, I'm a bit out of touch with many of the things that .NET offers and so it was simple ignorance on my part. Although I was extremely resistant because of previous experience, you taught me the error of my ways Thanks for your patience and tenacity on that matter but now you call it "stupid". The reason why I took such a stand, and I explained it on that thread, is that I have previously seen the "tower of babel" in real code and it wasn't simple ignorance on my part there. No one could maintain the bloody code because so many different languages were used and everyone that knew those languages well enough to be practical were gone from the company. Now THAT was both stupid and incorrect code! Worse yet, the code was in desperate need of modification because it could only get one piddly 30k row, 200 column wide file ready for import, never mind actually import it, in 45 minutes.
On the eBay thing, it would be both interesting and educational to further our discussion there but I suspect that we each have bigger fish to fry especially since I make no claims to being a front-end coder and, even if I was an expert there, would have little opportunity to test on the giant systems that eBay has. I do find it interesting that they did make a (in their words) a "small software change" back in 2013 that allowed them to take 400 servers (consumed a megawatt of power) out of their data centers and save 2 Million dollars. One wonders what else they could save on if they changed their tactics a bit more.
https://www.forbes.com/sites/michaelkanellos/2013/03/21/ebays-mpg-for-data-centers/#22521c829e02
https://readwrite.com/2013/03/11/new-ebay-metrics-help-save-millions-in-data-center-costs/
And a serious "Thank You" for the pleasant come back on the lucky part. We do bust a hump to prevent problems. More than a year ago, I had identified a single bit of front end code that issued a single bit of SQL code against the server and told management that we needed to fix it "now". It was being called thousands of times per hour, took 2 to 22 seconds to compile, and had to compile every time it was called because of the way the ORM generated the code. The front end developers wrote a stored procedure and modified the front-end code to use it but management delayed it because "It will take too long to test and we have other priorities". And so the code sat in waiting for several releases.About 5 months ago, we had a hell of a problem. The server is the "main" server for a lot of what we do and its where about 90% of the front end code works against for the floor of 400 or so Customer Service Reps and all external customers. It had 32 CPUs, 256GB of RAM, 3TB of SSD cache, and was running on the Enterprise Edition of SQL Server 2012. The code that I had identified the year prior and that had an already rewritten and unit tested fix waiting for testing reached a tipping point and drove all 32 CPUs to 80-85% usage and 19 of them were uber busy doing rollbacks. I told them we needed to do some quick emergency testing and get the fix in NOW! In return, I was told that was impossible because of other priorities and, besides, "Its a best practice to add hardware in such cases". <facepalm><headdesk> And so they overnighted 12 additional core of CPU and enough RAM to take us to 384GB. Do you have any idea what the cost of that is and what the SQL Server licensing costs are to do that? I told them it would not have any effect on the problem and could actually make it worse because more CPUs would dedicate themselves to doing rollbacks.
This is what it looked like during one of the episodes even after the additional 12 core. The width of the chart is 30 minutes. It was like watching a fire burning and someone had taken away my fire fighting equipment.
The heavy Black line is the CPU line with the "highlighter" turned on. The areas to the left and right of the big hump in the middle are what "normal" CPU had become. The heavy Red lines are long term blocking due to the rollbacks.One of the folks with some excellent Dev experience took all the symptoms I provided and Googled like hell for an answer. He found it and discovered that MARS had defaulted to ON for all the connection strings (which is contrary to the documentation on the subject) and that was the reason for the KILLED/ROLLBACKs I had discovered. All connection strings were changed to turn MARS off and that prevented the major "humps" like you see in the chart above but we still had the 18 to 22% CPU thing going on. Management finally made the "push" to do emergency testing on the fix we'd made months prior and suddenly we were back down to truly normal levels of CPU.
... and that includes both a document management system that was written in a silly manner and a large Web Methods job that ran.I am, indeed, very lucky to be working with such a great bunch of guys.
--Jeff Moden
RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.Change is inevitable... Change for the better is not.
Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally) -
At the QCON 2014 conference there was a very senior guy who said that he detested the term best practice because it implies that it's as good as it's going to get and there's no further improvement possible. He maintained that in the fast moving world of IT we always have to be open to the idea that improvement is not only possible but the driving force behind IT.
I think it might have been Tim Lister https://qconlondon.com/london-2014/london-2014/speaker/Tim+Lister.html
I think it is always important to publish the experiment, the result and the conclusions. All are up for critique but the critique itself must be supported by hard evidence that can be verified by the recipient of the critique. It's really tough having something you've worked on so hard for so long coming under criticism. Even when couched in diplomatic terms - David.Poole - Sunday, November 26, 2017 3:46 AM
Well I guess we could always go with "close enough for government work"
-
On the "tower of babel" thing... no... it wasn't stupid. As you say, I'm a bit out of touch with many of the things that .NET offers and so it was simple ignorance on my part. Although I was extremely resistant because of previous experience, you taught me the error of my ways Thanks for your patience and tenacity on that matter but now you call it "stupid". The reason why I took such a stand, and I explained it on that thread, is that I have previously seen the "tower of babel" in real code and it wasn't simple ignorance on my part there. No one could maintain the bloody code because so many different languages were used and everyone that knew those languages well enough to be practical were gone from the company.
Now THAT was both stupid and incorrect code! Worse yet, the code was in desperate need of modification because it could only get one piddly 30k row, 200 column wide file ready for import, never mind actually import it, in 45 minutes.I still think it is stupid, but I'd be happy to be advised of a more diplomatic description. I do admit to being frustrated with you as a poster, but I shouldn't get so carried away, I should rather just skip to the next post. Had you rather talked about the cost of adding each additional language, then THAT would have been a smart comment and could have started a very interesting conversation. on the other hand, if you COULD NOT FRAME the situation in such language, then yes, I regret your participation in that thread and even worse, you wouldn't budget on your assertion even when told VB was not even being used, it was a library routine being linked to and it was in the vb namespace. Maybe a better term would have been "obstinance?"
I told them we needed to do some quick emergency testing and get the fix in NOW! In return, I was told that was impossible because of other priorities and, besides, "Its a best practice to add hardware in such cases".
I continue to believe that saying something is a best practice doesn't make it a best practice. In fact, in your case, the advice followed such a familiar theme that it has its own catch phrase, "throwing hardware at the problem".
- patrickmcginnis59 10839 - Saturday, November 25, 2017 2:30 PM
The problem with best practices is people apply them blindly and that can be just as bad as not using any standards at all. And once you get to a higher level of development you are often times building things that no one has done before and there simply is no best practice to follow.
- patrickmcginnis59 10839 - Monday, November 27, 2017 6:29 AM
To be honest, I thought you already understood the costs involved in each language.
Shifting gears a bit, you do have some of the very diplomacy problems that you complain that other people on these have.
--Jeff Moden
RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.Change is inevitable... Change for the better is not.
Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally) - Jeff Moden - Monday, November 27, 2017 10:35 AM
Well I felt that discussing the costs of each additional language was a valid discussion. What's the additional language, does it bear any relation to the current set of languages that its being added to, what sort of contexts like the size of the organization or department tasked with supporting it, does the inclusion bring benefits that outweigh the costs, etc... and yes, others in that thread understood the costs so you're correct on that one.
But yeah, diplomacy, I could do better I'm sure! I'll definitely have to work on that one, especially here. Is "obstinate" ok?
- ZZartin - Monday, November 27, 2017 9:07 AM
I can agree with that. Folks applying anything with the title "best practices" blindly are just as bad as folks cutting and pasting any code from the web they find without vetting the suitability of their actions.
Viewing 12 posts - 31 through 41 (of 41 total)
You must be logged in to reply to this topic. Login to reply