Inside SQL Server

SQLServerCentral.com had the opportunity to talk to Microsoft and get a few questions about the upcoming release of SQL Server 2005 answered. We spoke with Tom Rizzo, Director, SQL Server and Alan Ros, Testing Manager for SQL Server.

On the CTP Process

SSC: It was announced on Apr 18 that there would be no Beta 3 for SQL Server 2005 and instead the CTP process would continue until the product RTMs. Can you give some background on why decision was made?

Thomas Rizzo We heard feedback from customers that they like the CTP process. More frequent drops of the product allows them to see incremental improvements across the entire product such as the components, documentation, samples, etc. Plus, moving to CTPs is a testament to where we are with the product. We feel good about our quality and do not need another major beta but instead move to more frequent CTPs for fit and finish work.

SSC: How has the CTP process affected the way development proceeds with SQL Server?

Thomas Rizzo It’s business as usual. CTPs are inline in our process. We actually get much better feedback from customers and partners by doing CTPs then doing Betas with long lag time between the Betas.

SSC: Does feedback from the CTP process give you a better idea of how close you are to RTM than the Betas?

Thomas Rizzo Yes. We can get feedback faster and customers can deploy and test more often. CTPs are a virtuous cycle between the SQL development team and our customers/partners.

SSC: When will performance tuning become a major factor in the CTPs?

Thomas Rizzo We’re already doing performance tuning and will continue to do it all the way until we release.

SSC: For the next release of SQL Server, do you anticipate any Betas at all?

Thomas Rizzo We’re experimenting here with the CTP process. Will this be the standard way we do this in the next release, it’s too early to tell but all signs are that the CTP process, whether it’s all the way throughout or at the tail end of the release, are all positive on doing CTPs at some point in the development process.

SSC: Are all Microsoft products moving towards the CTP process?

Thomas Rizzo Many are. It’s a great way to get technology into our customers hands and let them provide rich feedback.

SSC: If you receive quite a bit of feedback regarding a feature, good or bad, could it removed or added to SQL Server 2005 or is the product feature complete at this time?

Thomas Rizzo It’s feature complete. You won’t see major enhancements but instead fit and finish sorts of things.

SSC: When are the CTPs going to be up on the MSDN subscriber downloads? (Steve: I was joking here since the download wasn't up there Monday afternoon)

Thomas Rizzo They’re there today [Tuesday]. Download the April CTP and enjoy. While you’re enjoying the product, make sure to provide feedback as well! One interesting thing that many customers do not know is that you can report bugs online with SQL Server 2005. So if you run into an issue, make sure to go to http://lab.msdn.microsoft.com/productfeedback

Testing SQL Server

SSC: How many people are involved in testing SQL Server? (People exclusively testing as opposed to developers unit testing)

Alanros A lot!!! (Steve: hundreds, but we can't disclose the actual number)

SSC: What's the infrastructure like for testing?

Alanros There are actually several infrastructures. But the characteristics of them are the same. Every group has the ability to start a run automatically on any build and have tests executed automatically against that build. Most of them are able to do this from a single console or even have a job waiting for the drop to occur. The framework will handle spreading the tests out across all the machines that are used during that run. Example: In the Engine lab on Friday we had 12 different lab runs which were all started via the automation and executing on 500 machines.

SSC: There is a video on Channel 9 regarding the "" testing tool. Are there any other tools that you use for testing?

Alanros This is just scraping the surface. ARUBA which is discussed in this video is one verification tool used by one group. There are unique tools in almost every group that help to perform testing or verify the results of testing in an automated fashion. For instance in the engine group we have a stress driver that emulates many active users and causes a workload to be run. We also have the stress driver running background workloads such as database backups at the same time. Bugs that are hit are then automatically filed in the bug database. We can have at least 20 machines running these tests automatically and filing bugs with no human intervention. Other groups also have state machine modeling tools to help generate test cases automatically. We also have the ability to take a user database schema and statistics and recreate a database with similar distribution of data and run automatically generated queries against that data.

SSC: What type of scalability testing do you do? Are the CTPs tested anywhere near the TPC levels?

Alanros We are continuously running all the benchmarks and performance tests throughout the process so by definition the CTPs will be at the highest levels available at the time we ship them. The same is true for our scalability tests such as making sure features that should be able to use multi-proc or have parallelism in them do indeed achieve those. I believe most folks view CTPs as something off to the side but it really is an inline process that we work on as we head for our RTM.

SSC: What type of reliability/stability testing do you do and how do you measure this?

Alanros Every group does either long haul testing or stress testing on our way to shipping the CTP or the product. Some are measure how long they have been up and running and these are known as long haul testing. They must execute properly of course and generate no AVs (Access violations) or ASSERTs over a pre-defined length. The length can be anywhere from 48 hours up to a week. It depends on the component. The second method is to have a number of stress tests that are continuously executed one behind each other. These tests usually have a predetermined runtime with a predetermined number of users. They are executed on all platforms that we are supporting. These also look for AVs and ASSERTs and perform database checks at the end to insure we have a valid database.

SSC: Does the CTP process create problems with testing SQL Server?

Alanros Not really. As a matter of fact as I stated before, we view CTPs as an inline process on our way to RTM. It is a way to make sure we measure the increase in quality that we want as we approach RTM. The release criteria we assign to each CTP is always increasing.

How much do the developers participate in the testing process?

Alanros This depends on the group. Developers write unit/check in tests, execute check in tests prior to check in, run BVTs on their branches prior to putting code into general branches for general consumption.

SSC: Do the CTPs get a full regression test of items to date before release?

Alanros Yes

SSC: How much does a regular release cycle, say an every other month CTP, impact your testing?

Alanros I look at the delivery of putting a CTP out as a mechanism to insure we are tracking to our RTM requirements. Yes, it does take additional work but it truly is not excessive and yet we can achieve our quality goals.

SSC: How do you decide which bugs/issues require more development and which are released in a CTP with feedback to the development team?

Alanros We look at several factors when determining to fix a bug. 1) Is the bug mainstream or not, i.e. will many customers hit the issue. 2) The severity of the error. Something like just word changes in an error message will not get much attention. But of course AVs or data loss reported by our customers are fixed. 3) What is the impact to the customer when they see the issue? 4) Where are our quality bars and what bugs are preventing us from reaching those quality bars.