December 26, 2011 at 10:12 am
Since I also previously asked all of you good folks to not publish an article on the rapid generation of test data, I have to ask... do any of you see or have a problem with me writing and publishing an article on how to rapidly build test data?
No problem, in fact have marvelled at your hesitation in the past to do so.
December 26, 2011 at 10:17 am
Can't understand why you're asking Jeff - any thing like that would be appreciated by old and new hand alike IMHO
-------------------------------Posting Data Etiquette - Jeff Moden [/url]Smart way to ask a question
There are naive questions, tedious questions, ill-phrased questions, questions put after inadequate self-criticism. But every question is a cry to understand (the world). There is no such thing as a dumb question. ― Carl Sagan
I would never join a club that would allow me as a member - Groucho Marx
December 26, 2011 at 12:42 pm
Thanks for the feedback. Yeah, I know. I sound like a real wire-head trying to explain why I didn't want to publish such a thing and it's a really hard thing to explain to begin with. I'll just leave it at that.
Considering your replies, Bob's feedback, and the feedback I've gotten from DBAs that don't frequent SSC, it's time for me to write the article on test data generaton and publish it.
Thanks for your time and Merry Christmas. 🙂
--Jeff Moden
Change is inevitable... Change for the better is not.
December 26, 2011 at 12:47 pm
Jeff Moden (12/26/2011)
Thanks for the feedback. Yeah, I know. I sound like a real wire-head trying to explain why I didn't want to publish such a thing and it's a really hard thing to explain to begin with. I'll just leave it at that.Considering your replies, Bob's feedback, and the feedback I've gotten from DBAs that don't frequent SSC, it's time for me to write the article on test data generaton and publish it.
Thanks for your time and Merry Christmas. 🙂
Merry Christmas to you ... hope you do not feel that you took down your stocking from over the fireplace and our comments were all it contained and the "lump of coal"
December 26, 2011 at 1:00 pm
bitbucket-25253 (12/24/2011)
Lowell your posting of this link to assist an OPTallyCalendar_Complete_With_DST.txt
Lowell -Fantastic amount of work - from a IT standpoint thanks for sharing - Glad you were willing to play Santa and give me this present.
For all other denizens of the thread - read Lowell's work it will be a great present for most of us.
Thanks Ron and everyone else who thinks they can use it;
if you look it over, it's a collection of various snippets from right here in the forums...the obvious Tally Calendar reference that I built off of one of Jeff Moden's many contributions, plus a bunch of stuff I added myself;
I know I added a lot of the calculations for dates myself as I made myself intimately familiar with the date add functions....
somewhere out there is the thread where folks helped me convert my scalar calculate-lunar-period to an ITVF.
another thread, someone else had the DST rules for America for before 2006 vs after, but needed help applying it in the thread; i added that too.
Just recently Remi (Ninja) pointed out the advantages of having an ISO year-month column, and other examples as well.
I think it's at least a good snippet to keep in the toolbox, if not a solid add-on for whatever Tally Calendar equivalent you guys use now.
Thanks for the tip of the hat!
Lowell
December 26, 2011 at 3:25 pm
Jeff Moden (12/26/2011)
I have a serious question for my fellow Threadizens...... do any of you see or have a problem with me writing and publishing an article on how to rapidly build test data? My purpose in doing such a thing is to make it a bit easier for all of us to say, "Your claims of performance are unsubstantied without scads of test data. Please see the following article for how to do so and why you must do so."
We really need to have it finally done. I know I would benefit from it. I would have no problem with you doing that.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
December 26, 2011 at 7:30 pm
SQLRNNR (12/26/2011)
Jeff Moden (12/26/2011)
I have a serious question for my fellow Threadizens...... do any of you see or have a problem with me writing and publishing an article on how to rapidly build test data? My purpose in doing such a thing is to make it a bit easier for all of us to say, "Your claims of performance are unsubstantied without scads of test data. Please see the following article for how to do so and why you must do so."
We really need to have it finally done. I know I would benefit from it. I would have no problem with you doing that.
I can be biased, but I would make sure that the methodology conforms to http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/ref=sr_1_1?ie=UTF8&qid=1324952533&sr=8-1.
If it does not - and it may, that's allright - just please explain why it does not and what makes your tack better.
Please understand, I will be the first one to applaud if you come up with something better, and I will be proud to evangelize any new, better approach.
December 26, 2011 at 8:02 pm
Jeff Moden (12/26/2011)
A couple of years ago, one of our own asked me if he could publish an article on how to build scads of test data. The reason why he asked me is because of the methods I use to build my usual "million row test table" for most of my articles and posts. Although the methods certainly aren't my idea alone, I asked that person and, in the process, everyone on this thread to not publish an article on how to quickly generate a large test table regardless of the method used. My reasoning was that being able to quickly create a million rows of test data separated the proverbial "men from the boys" when it came to writing code and that I preferred for "contributors" (folks that partook in reading our articles and posts) to learn it by actually participating in the forums rather than handing it to possible "non participants"
I think I have to disagree with you a bit on this one Jeff.
My reason will pretty obvious, I think, to people who know my background. Part of my jobs,pretty well throughout my career, has been to teach - and although my ideal has always been to teach people how to think for themselves, how to work things out for themselves, I've also found it necessary to teach people what to look for, and to use examples to show how it's done. I think the post you are responding to is a pretty good demonstration of how it's done. Without posts like that people won't learn. The people who have any chance of ever being good will study it closely and work hard to understand why this is the right way (or the wrong way, if they disagree with it, and those may be the ones who learn most from it) to do it. The people who just copy it and use it without bothering to understand it would never have been any use anyway, so it does no harm (except perhaps to interviewers and recruitment agents who can't distinguish parrots from people, of whom I have met far too many).
I don't like disagreeing with you, because I have learnt from reading your articles that you are far brighter and far more knowlegeable about T-SQL than I am. But this time I think you are wrong - maybe only a little bit wrong, but wrong nevertheless.
Tom
December 26, 2011 at 8:39 pm
Revenant
I can be biased, but I would make sure that the methodology conforms to http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/ref=sr_1_1?ie=UTF8&qid=1324952533&sr=8-1.
Gosh - if that is the definitive method then Lord help us all. But I will say one thing, when MS tests they may catch the small things, but the real biggies, hmm shall we start to list the disasters and weakness of MS software. Nah do not have the time to sit and list them all, but for grins and giggles go to Microsofts own web site
http://connect.microsoft.com/intro/
and find the following
To date, more than 90,000 defects have been fixed
. True that number was tracked beginning in July 2005, so lets see that is 15,000 defects per year missed by the suggested testing methodolgy.
Yeah that speaks ever so highly of whatever is contained in that reference.
December 27, 2011 at 5:21 am
bitbucket-25253 (12/26/2011)
Revenant
I can be biased, but I would make sure that the methodology conforms to http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/ref=sr_1_1?ie=UTF8&qid=1324952533&sr=8-1.
Gosh - if that is the definitive method then Lord help us all. But I will say one thing, when MS tests they may catch the small things, but the real biggies, hmm shall we start to list the disasters and weakness of MS software. Nah do not have the time to sit and list them all, but for grins and giggles go to Microsofts own web site
http://connect.microsoft.com/intro/
and find the following
To date, more than 90,000 defects have been fixed
. True that number was tracked beginning in July 2005, so lets see that is 15,000 defects per year missed by the suggested testing methodolgy.
Yeah that speaks ever so highly of whatever is contained in that reference.
15000 per year is just the ones they've fixed; maybe it's more like 20000 per year, allowing for the things they don't fix.
But don't be too harsh on the testing method - there is no such thing as a testing method that will find all faults in complex software.
Tom
December 27, 2011 at 6:26 am
It gives me immense pleasure to declare that I am able to capture a very precious moment in SSC 🙂 (enclosed here). I acquired proficiency level ‘Default Port’ today & enjoyed it for more than 1 hour.
For those who are unaware of ‘Default Port’ status, it’s valid only for Point#1433. It’s momentary thus very precious than any other status. I planned for it when I came to know about it & was continuously monitoring my progress to capture it. :Whistling:
I am also feeling bad to post this acknowledgement because I know when I will be done I will lose it. :unsure:
December 27, 2011 at 6:37 am
Jeff Moden (12/26/2011)
The Question: (well, sort of :-))... do any of you see or have a problem with me writing and publishing an article on how to rapidly build test data? My purpose in doing such a thing is to make it a bit easier for all of us to say, "Your claims of performance are unsubstantied without scads of test data. Please see the following article for how to do so and why you must do so."
Any and all feedback (even if it's an ad hominem attack, rant, or volley of high velocity porkchops... yeah... I'm really that interested in your thoughts. Just save some of the porkchops for the folks making unsubstantiated claims of performance :w00t:) on the subject of writing such an article would be greatly appreciated.
Go for it.
Make sure you include methods of generating a variety of selectivity scenarios, since real testing requires not just lots of rows, but variations in selectivity/cardinality/et al.
I'm definitely interested in seeing how your methods differ from mine, and how they overlap.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
December 27, 2011 at 8:23 am
Jeff Moden (12/26/2011)
I have a serious question for my fellow Threadizens...
Based on the fact that we can't get people to run good sets of backups despite Umptymillion articles, blog posts, presentations, webcasts and screaming lightning talks, I don't think you're going to be giving away the secrets of the universe. In fact, I think you could publish the secrets of the universe and they'd still be mostly safe because of a serious inability to RTFM/A/B by most people. Heck, I'd say that 1 in 20 actively resists fundamental advice "Yeah, that WHILE loop that you're using to load a million rows inside of a multi-statement UDF that gets joined to another multi-statement UDF with a WHILE loop... that's your problem" "Oh, well, I don't want to change that. What else can we do?"
In short, go for it. It'll prove useful to those who care and can actively learn from it.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
December 27, 2011 at 9:35 am
L' Eomot Inversé (12/27/2011)
bitbucket-25253 (12/26/2011)
Revenant
I can be biased, but I would make sure that the methodology conforms to http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/ref=sr_1_1?ie=UTF8&qid=1324952533&sr=8-1.
Gosh - if that is the definitive method then Lord help us all. But I will say one thing, when MS tests they may catch the small things, but the real biggies, hmm shall we start to list the disasters and weakness of MS software. Nah do not have the time to sit and list them all, but for grins and giggles go to Microsofts own web site
http://connect.microsoft.com/intro/
and find the following
To date, more than 90,000 defects have been fixed
. True that number was tracked beginning in July 2005, so lets see that is 15,000 defects per year missed by the suggested testing methodolgy.
Yeah that speaks ever so highly of whatever is contained in that reference.
15000 per year is just the ones they've fixed; maybe it's more like 20000 per year, allowing for the things they don't fix.
But don't be too harsh on the testing method - there is no such thing as a testing method that will find all faults in complex software.
This is how the numbers work:
According to Microsoft, industry grade code has no more than one bug per 50,000 net lines of code.
The challenge is that Windows is about 50 million lines of code and SQLS 2008 R2 about 20 million. (I don't know the number for 2012 yet.)
On each new build of SQLS, 230 quite powerful VMs are running thousands of tests for several days, close to a week.
These numbers may be impressive, but guys in Building 35 are still looking for more test benches and better methods. If any of you have some ideas, I am positive they will listen.
December 27, 2011 at 11:14 am
bitbucket-25253 (12/26/2011)
Revenant
I can be biased, but I would make sure that the methodology conforms to http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/ref=sr_1_1?ie=UTF8&qid=1324952533&sr=8-1.
Gosh - if that is the definitive method then Lord help us all. But I will say one thing, when MS tests they may catch the small things, but the real biggies, hmm shall we start to list the disasters and weakness of MS software. Nah do not have the time to sit and list them all, but for grins and giggles go to Microsofts own web site
http://connect.microsoft.com/intro/
and find the following
To date, more than 90,000 defects have been fixed
. True that number was tracked beginning in July 2005, so lets see that is 15,000 defects per year missed by the suggested testing methodolgy.
Yeah that speaks ever so highly of whatever is contained in that reference.
All things considered, the software Microsoft produces works amazingly well.
Linux and OSX both have years of work to go to catch up with Windows in terms of security, stability, scalability, and hardware support. For example, of all the computers I own, the one running Linux is the one that has an official recommendation that I reboot it at least weekly. In June of this year, OSX finally caught up with Windows XP SP2 in security and stability. Yes, OSX is finally as secure as an OS that's been out of support for several years. (Not per Microsoft, but per review by NSA and security companies.)
The idea that Microsoft produces buggy, insecure, crash-prone systems, these days, is primarily an artifact of the marketing done by their competition.
Don't buy the marketing hype from Apple, Oracle, or Linux fans. They have a strong vested interest in FUDing Microsoft. Just as strong as MS's interest in FUDing them. Don't fall for the propaganda from either side of that battle.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
Viewing 15 posts - 32,896 through 32,910 (of 66,712 total)
You must be logged in to reply to this topic. Login to reply