May 12, 2015 at 12:44 pm
I'd love to hear what Microsoft do with testing SQL Server. Do they use TDD/BDD, manual methods and are they designing SQL Server code to be testable?
What are the corrective actions that took place to ensure that the root cause doesn't happen again?
I think it would make a good product article from a Microsoft SQL Server team member.
May 12, 2015 at 6:10 pm
David.Poole (5/12/2015)
I'd love to hear what Microsoft do with testing SQL Server. Do they use TDD/BDD, manual methods and are they designing SQL Server code to be testable?What are the corrective actions that took place to ensure that the root cause doesn't happen again?
I think it would make a good product article from a Microsoft SQL Server team member.
Or a very, very bad one. :ermm:
--Jeff Moden
Change is inevitable... Change for the better is not.
May 13, 2015 at 2:57 pm
Jeff Moden (5/11/2015)
MS has to do one very important thing... they need to slow down and do service packs 100% correct. SPs are supposed to fix things, not break them or send you back two revs by mistake.
They need to get their service packs a lot better. Certainly - no disagreement there.
But... If they were capable of getting things 100% correct they could do that one releases, not just on service packs. And the we wouldn't need service packs, would we, unless service packes were intended not to fix problems ut th add new features between releases.
So I think 100% correct service packs is a crazy idea - but the closer MS get to that the happier I will be.
Tom
May 13, 2015 at 3:43 pm
TomThomson (5/13/2015)
Jeff Moden (5/11/2015)
MS has to do one very important thing... they need to slow down and do service packs 100% correct. SPs are supposed to fix things, not break them or send you back two revs by mistake.They need to get their service packs a lot better. Certainly - no disagreement there.
But... If they were capable of getting things 100% correct they could do that one releases, not just on service packs. And the we wouldn't need service packs, would we, unless service packes were intended not to fix problems ut th add new features between releases.
So I think 100% correct service packs is a crazy idea - but the closer MS get to that the happier I will be.
It's a funny thing. Not just here but everywhere (except where I currently work) that I suggest that someone test things well enough to be fault free and have no regression errors, I get told it's a crazy idea. 😛
--Jeff Moden
Change is inevitable... Change for the better is not.
May 14, 2015 at 12:20 pm
Jeff Moden (5/13/2015)
TomThomson (5/13/2015)
Jeff Moden (5/11/2015)
MS has to do one very important thing... they need to slow down and do service packs 100% correct. SPs are supposed to fix things, not break them or send you back two revs by mistake.They need to get their service packs a lot better. Certainly - no disagreement there.
But... If they were capable of getting things 100% correct they could do that one releases, not just on service packs. And the we wouldn't need service packs, would we, unless service packes were intended not to fix problems ut th add new features between releases.
So I think 100% correct service packs is a crazy idea - but the closer MS get to that the happier I will be.
It's a funny thing. Not just here but everywhere (except where I currently work) that I suggest that someone test things well enough to be fault free and have no regression errors, I get told it's a crazy idea. 😛
No, testing things so that they are 100% correct is not a crazy idea. The crazy idea is that once you achieve that you sytill need service packs. Presumably they correct the errors you didn't make? :w00t:
Tom
May 16, 2015 at 10:39 pm
TomThomson (5/14/2015)
Jeff Moden (5/13/2015)
TomThomson (5/13/2015)
Jeff Moden (5/11/2015)
MS has to do one very important thing... they need to slow down and do service packs 100% correct. SPs are supposed to fix things, not break them or send you back two revs by mistake.They need to get their service packs a lot better. Certainly - no disagreement there.
But... If they were capable of getting things 100% correct they could do that one releases, not just on service packs. And the we wouldn't need service packs, would we, unless service packes were intended not to fix problems ut th add new features between releases.
So I think 100% correct service packs is a crazy idea - but the closer MS get to that the happier I will be.
It's a funny thing. Not just here but everywhere (except where I currently work) that I suggest that someone test things well enough to be fault free and have no regression errors, I get told it's a crazy idea. 😛
No, testing things so that they are 100% correct is not a crazy idea. The crazy idea is that once you achieve that you sytill need service packs. Presumably they correct the errors you didn't make? :w00t:
Perhaps that's when DBCC TIMEWARP comes into play... you fix the mistakes you haven't made yet?
--Jeff Moden
Change is inevitable... Change for the better is not.
May 17, 2015 at 3:08 am
Jeff Moden (5/16/2015)
TomThomson (5/14/2015)
Jeff Moden (5/13/2015)
TomThomson (5/13/2015)
Jeff Moden (5/11/2015)
MS has to do one very important thing... they need to slow down and do service packs 100% correct. SPs are supposed to fix things, not break them or send you back two revs by mistake.They need to get their service packs a lot better. Certainly - no disagreement there.
But... If they were capable of getting things 100% correct they could do that one releases, not just on service packs. And the we wouldn't need service packs, would we, unless service packes were intended not to fix problems ut th add new features between releases.
So I think 100% correct service packs is a crazy idea - but the closer MS get to that the happier I will be.
It's a funny thing. Not just here but everywhere (except where I currently work) that I suggest that someone test things well enough to be fault free and have no regression errors, I get told it's a crazy idea. 😛
No, testing things so that they are 100% correct is not a crazy idea. The crazy idea is that once you achieve that you sytill need service packs. Presumably they correct the errors you didn't make? :w00t:
Perhaps that's when DBCC TIMEWARP comes into play... you fix the mistakes you haven't made yet?
If you think about how you define 100% correct, then it becomes obvious that you are both right.
100% success on testing does not equal 100% coverage on testing.
If I only test for 3 out of 5 possible scenarios, and they all pass, I am at 100% success, until someone discovers the other 2 tests that I never ran. 😉
MM
select geometry::STGeomFromWKB(0x0106000000020000000103000000010000000B0000001000000000000840000000000000003DD8CCCCCCCCCC0840000000000000003DD8CCCCCCCCCC08408014AE47E17AFC3F040000000000104000CDCCCCCCCCEC3F9C999999999913408014AE47E17AFC3F9C99999999991340000000000000003D0000000000001440000000000000003D000000000000144000000000000000400400000000001040000000000000F03F100000000000084000000000000000401000000000000840000000000000003D0103000000010000000B000000000000000000143D000000000000003D009E99999999B93F000000000000003D009E99999999B93F8014AE47E17AFC3F400000000000F03F00CDCCCCCCCCEC3FA06666666666FE3F8014AE47E17AFC3FA06666666666FE3F000000000000003D1800000000000040000000000000003D18000000000000400000000000000040400000000000F03F000000000000F03F000000000000143D0000000000000040000000000000143D000000000000003D, 0);May 17, 2015 at 6:57 am
Agreed. I guess that's what I'm griping about. They don't actually know how to test for what or that "what" even exists.
--Jeff Moden
Change is inevitable... Change for the better is not.
May 17, 2015 at 6:28 pm
mister.magoo (5/17/2015)
If you think about how you define 100% correct, then it becomes obvious that you are both right.
Of course we're both right. Jeff is never wrong. I'm only wrong when either my wife says so (she never talks about software) or I'm talking through my hat. I'm currently in the UK and I left all my hats in Spain, so .... ...
Tom
May 17, 2015 at 7:13 pm
Jeff Moden (5/17/2015)
Agreed. I guess that's what I'm griping about. They don't actually know how to test for what or that "what" even exists.
A big part of the problem (I think) is the tendency to move responsability for testing away from development, and employ test/QA people who have no idea what the software is supposed to do other that what they can get from a draft user manual. Another big part is management that imposes loony timescales and insists on stuff going out of the door on time even if the late development means that QA only has 3 days instead of 3 months to check the stuff (and they won't be allowed to reject it, because the release date is wriiten in stone). Another problem is developers who think all testing should be done by someone else, so that unit tests are pretty meaningless (if they exist at all).
But that definitely leaves a large part that is caused by people not knowing how to test, not knowing what to test, not bothering about where there are boundary conditions that may cause an unexpected change in behaviour. Developers can make that last mistake as well as testers/QA people, and so can DBAs. And non-technical managers make it too.
I don't actually see a solution. If we could all adopt semi-formal methods (as IBM did for CICS, where there was a serious collaboration with Oxford's PRG to use a combination of Z and Dijkstra guarded commands in the development of CICS/ESA 3.1) we could (a) detect more problems independently of testing and (b) have a better idea of what needs testing - but the publicly available documentation of that project is pretty sparse, and some of the benefits IBM reckons it got (9% development cost reduction, and a factor of more that 2 reduction in customer bug reports) have perhaps been exagerated by IBM and by OXPRG (designers of rival formal systems are, of course, almost unanimously agreed that they are). But the IBM/OXPRG collaboration didn't include formal methods for verification, only formal methods for development, so testing adequately would still remain a serious issue even if everyone adopted the methods used there. As far as I can tell, every attempt to use formal methods for verification has failed - but I'm out of touch with latest develpment in the field, so maybe there's some light at the end of the tunnel now.
Tom
May 18, 2015 at 1:45 am
This post seems to have morphed from MS producing crap SPs to a discussion on testing (not necessarily a bad thing)
Here's my two pennethworth - QA/testing should be the police - if it don't pass it don't ship. Dev who don't test don't work. Management take responsibility - businesses that ship crap owe their customers. DBAs are responsible for their business continuity - they need faith in what they install
May 18, 2015 at 3:17 am
A big part of the problem (I think) is the tendency to move responsability for testing away from development, and employ test/QA people who have no idea what the software is supposed to do other that what they can get from a draft user manual.
Very much hit the nail on the head. Anyone who voluntarily abdicates responsibility for testing to the testers is asking for trouble. Testing is very much a skill. The best ones are those who can look at a complicated use case and instantly spot a gap in the requirements.
I believe that testers should work closely with business analysts so they can help define how the software being defined should be tested. The testers should then work with the developers to help define how the developers can implement the testing.
A good developer will do their utmost to ensure their code does what they intend it to do but in the testing world at best they will be gifted amateurs playing to a craftsman's game.
I believe that engaging a test specialist early on results in a much clearer set of requirements and therefore games the delivery in favour of the developer and ultimately the quality of the resulting software.
May 18, 2015 at 1:54 pm
Fair points on testing, but I'll say that this isn't simple. With respect to Jeff and Tom, I don't agree with 100% of anything. At most any scale for software, it's simply impossible in any practical way to test everything. Especially when you sell software and can't conceive of the permutations in which the software might be installed and used.
That being said, certainly testing is not well done, though I'd argue it's never well done in the history of software. CICS certainly still had issues, though I bet as a percentage of code points they had less.
I haven't seen QA removed from testing as much as I've seen QA embedded into testing, which of course, is a different problem. As developers, we tend to test the happy path and look for things to function as we expect. We add "tests" (used very loosely, as this lacks engineering just as the development process lacks engineering), when we get bugs reported or problems ensue. That means we're chasing our old work, with new work coming.
AFAIK, Microsoft does write lots of tests on the SQL Server product. Hundreds of thousands of them, which are run. However these verify that code does what it should, or returns errors as it should. When we combine multiple units together, there are missing tests. There are ways in which SQL Server gets installed that people at Microsoft never anticipated. In fact, I run into people all the time with configurations that I'd never expect.
Microsoft takes the product after it's passed automated tests and there are QA people looking at other items, which further catches things, but still obviously has issues.
If you want to tell me that you manage 100% testing in some way, then you should be showing that in the last year or so you have no written any code that gets returned to you with an issue. No matter what the data, no matter how it's run.
If you can do that 100% of the time, then maybe you have teach the world about testing. However as often as I may write code that works great and isn't returned, I always find there are some items that don't perform as expected under all conditions.
May 18, 2015 at 2:22 pm
I heard a story about a team that absolutely positive that they had got a test for every possible permutation and combination of inputs that could be thrown against their software. They called in Mr super tester who simply rested a book on the keyboard and the system crashed.
If you write fool-proof software then the universe will invent a new type of fool. To anyone who thinks you can get 100% test coverage go and work for the UK Inland Revenue or whoever they are outsourcing their software development to these days. The tax system is so convoluted that an entire branch of accountancy has arisen that can function as a profitable stand alone business to help people interpret the rules.
100% testing is a worthy aspiration and we should all strive towards it in the same way that an Olympic athlete strives to beat world records. Just don't get too disheartened that we won't see a 3 minute mile.
May 18, 2015 at 6:23 pm
David.Poole (5/18/2015)
100% testing is a worthy aspiration and we should all strive towards it in the same way that an Olympic athlete strives to beat world records. Just don't get too disheartened that we won't see a 3 minute mile.
For something like an operating system or a large scale RDBMS 100% testing is more like a 3 second mile than a 3 minute one.
Tom
Viewing 15 posts - 16 through 29 (of 29 total)
You must be logged in to reply to this topic. Login to reply