Welcome to the Spotlight Behind SQL Server, a new series from SQLServerCentral.com. As we've grown and spent more time covering SQL Server, we've slowly gained a number of contacts inside Microsoft, including those that develop the product. And we decided to try and interview the SQL Server people inside Microsoft. There are lots of people working on SQL Server 2005 and our goal to is to eventually get to them all.

We know that there are lots of technical things we could ask, and lots of easy marketing questions we could get from them, but you probably read most of those questions elsewhere. So we thought we'd make them think a bit more and get some interviews that showcased the people behind SQL Server. To that end, these interviews will be a little bit different and give you a look at the amazing team that builds SQL Server.

We caught up with Bob Dorr recently after the PASS conference where he hosted a very interesting session on the internals of the SQL Server engine.

SSC : What's your official title and responsibility at Microsoft?

Bob : Senior SQL Server Escalation Engineer

SSC : What part of SQL Server 2005 did you see the most calls about?

Bob : Before SQL Server 2005, the database engine dominated the call volume for PSS. In prior releases we often found 1 or 2 bugs that would quickly focus and dominate the call volume until they were fixed. We have not seen this with SQL Server 2005. SQL Server 2005 quality has exceptional, and the engine calls are spread across many aspects of the product.

When you look at the sales volume compared to the call volume it is not a surprise as the call volume follows a standard industry curve. We made great strides in the BI stack (SSIS, OLAP, Report Services) and the volume in these areas has picked up significantly along with the sales of these systems.

We have taken a very prescript approach to addressing issues with SQL Server 2005. Combined with this the PSS and development teams attacked Dr. Watson reports and call code information. Each support case is coded (call coded) by symptom and the symptoms mapped to product areas. Each major area (called a pipeline) is assigned to a program manager, escalation manager and an escalation engineer: we call this the Red Zone. Monthly the pipeline owners from PSS and Development meet and identify product improvements, documentation needs, file DCRs and bugs, and take other necessary actions. Quarterly each pipeline reports out to the upper management of PSS and development and includes project timelines to address the pressing issues.

SSC : Is there one thing you wish all the DBAs out there knew to cut down on support calls

Bob : The #1 thing that continues to be a problem is backups. There is always that very clear line of distinction when talking to an admin that has lost data before and those that have only planned for data loss. Restoring them from time to time and actually attempting the data recovery scenario in a test environment would prevent a bunch of critical calls to us each month.

I think our current critical call volume for SQL Server is around 80 / month. Of those a good 40% or more are related to things like someone truncated a table and I need to get the data back or I had a drive failure and need to get the data back. They are generally self inflicted types of issues that only get worse when there is no backup.

The second thing is change tracking. Customers that implement strong change tracking controls call us far less than others, and when they call they are much more prepared to solve the problem. We can often get to the root cause much faster because the DBAs understand what has recently changed.

SSC : How early does PSS get involved in the SQL Server development?

Bob : This process has grown over the years. When I joined Microsoft in 1994 the SQL team was very small and had just completed the formal transition from Sybase. In those days it was not so much about new product features but making what we had work well. However, each successive product has incorporated PSS sooner and more often. Today both Bob Ward and I are primarily dedicated to working with the development team on current and new product feedback. The rest of the escalation team is also involved in their areas of specialty as well. We travel to Seattle several times and year, sit with the various developers, go over designs, ask questions, and challenge them. For SQL Server 2005 the entire US escalation team as well as many international escalation engineers completed 2 week internships as far back as 2002. Many of us participated in multiple internships over several years.

We actually review, comment and have sign-offs on various specification documents. For example, we have successfully introduced a supportability specification to the development process. It covers everything from error message context to tracing capabilities. PSS is a key stakeholder in signing off on this document, making sure the product is supportable by the customer and PSS.

Some more specific examples you might be familiar with are 17883, checksums, and stalled I/O warnings. Each of these involved a ton of PSS input and discussion with the development team. The development team is so good at helping us attack and resolve re-occurring problems. Customers were getting stalls and corruptions that PSS kept tracking to things outside the SQL Server. Microsoft studied the patterns and worked with the development team to put solutions in place that would quickly point out root cause of these types of problems. So much exchange occurred that the 17883 and stalled I/O warning was added as a service pack enhancement to SQL Server 2000 and did not wait for the SQL Server 2005 release.

SSC : Does the PSS team help prioritize which bugs will get patched in Service Packs and/or Hotfixes?

Bob : We have always done this but it has historically been some of the seasoned engineers bring up trends to be fixed.

It was around 2 years ago that the SQL CSS team hired a statistical engineer to assist us. As each case is closed we call code the case to the area/symptom of the product. She runs various analysis over this data and provides key details such as number of issues, average days it takes to solve, average labor it takes to solve, cost to Microsoft and so forth. This group also has way of associating symptoms with a customer pain index. This data is all combined with QFE data in these areas as well as Dr Watson reports to generate a master list.

The SQL white box is then subdivided into pipelines. For example (SSIS, OLAP, Engine, Performance, ) Each pipeline meets several times a month to discuss the statistical trends, QFE requests and cases that are open and difficult to solve. From this data a priority list of issues is identified and fix targets established.

This list is presented in joint quarterly business meeting between CSS and SQL Development. This meeting includes the director of SQL support, GMs of SQL support, Bob Ward and myself. It also includes the VP of SQL development, the product unit managers (PUMs) and the development pipeline owners. The meeting is referred to as the 'RedZone' meeting. This type of format is used with all our major products. Each group has to sign up for what they are going to fix along with the metric goals like reduce call volume, reduce cost to support and such things and progress is updated each quarter.

Outside the formal process the seasoned members of the team mark bugs for inclusion in a service pack as we find them in our day to day jobs. Then as we prepare for a service pack, folks like myself participate in joint bug triages with the development teams.

This allows us to take a business savvy approach but avoid the pitfall of only looking at statistical values and ignoring what we term as 'duh or no-brainer' fixes.

SSC : How large is the SQL Server support team in Texas?

Bob : The overall support site at Texas comprises two four story buildings. These house support, labs and sales. The SQL Server staffing in TX has been the largest site in the US since about 1998, but our Charlotte, NC site is usually neck and neck with us. The SQL Server support portion of this is about 40 people and about 1/2 of that is escalation staff. By far we have the largest escalation team in the world located in Texas.

SSC : How long have you been working on SQL Server?

Bob : I started working with SQL Server on OS/2 at State Farm in the early 90s. I was a member of the development team for the State Farm Catastrophe System (CAT). This was one of the first PC based divisions at State Farm. The system was designed using an OS/2 server running SQL Server and Windows for Workgroup clients. Written in C/C++ using the DOS, Panel screen library. It was all bundled together on pallets in a warehouse (paper, tables, PCs, printers, modems, tape backup systems, .) When a storm would hit someplace the system was dispatched. State Farm might take over a garage, set of hotel rooms or even a mobile trailer. The system was setup by the adjusters dispatched to handle the site.

We only had 8 developers for the entire system, so you worked on everything. I wrote assembly code to control a tape drive and batch load claim data. We did geo coding with a product called Map View that had to be mapped to the system. Tuned queries, handled backups, RAS interfaced with the regional mainframes and I even had to learn HP PCL printer sequences to handle draft and mail merge operations. I got a really good overview of a lot of technologies very quickly.

The team also had to provide support for our own system top to bottom. We did not have a separate helpdesk, we carried the pagers. I learned a lot about basic support needs, error message context and associated documentation for example. I learned a lot about good coding practices such as put the constant on the left side of an evaluation so an assignment mistake becomes a compile problem and not a runtime issue. I remember taking a class on screen standards and display that taught me about screen density, and the list goes on.

However, my passion was SQL Server and client server programming. This eventually lead me to Microsoft SQL Server support in Oct 1994. I progressed to a technical lead in 1996 and joined escalation in 1998. I have been thrilled with solving the hard problems and making the product better for customers ever since.

SSC : Give us a little background on yourself, how did you get into computers?

Bob : I have always been fascinated with computers and numbers. My parents owned a farm machinery dealership when I grew up. When I was in the 6th grade they got a Commodore (I think it was an 8510 but that was a long time ago). I remember it had a dual 5 1/4 floppy drive system top of the line for its time. My parents had a programming book and basic and I was printing my name (banner style repeated letters making up an 11 inch tall "B O B") on the old 9-pin printer that same night. I made my own games and helped my parents with inventory and VisaCalc programs.

When I got to high school I upgraded to my own computer. It was a Mac 8 or 9 inch black and white screen, all in one. I got my hands on the Lightspeed Pascal and C compilers and I was off and running. I went through upgrades and eventually got a color screen, I did everything on it. In fact, my high school English teacher once told me that I should always make sure I had a spell checker; I would need it.

I went to Midland Lutheran College in Fremont NE. I actually started on a Business major Accounting concentration. I started taking some computer classes DB2, Cobol and eventually found myself in an assembly class. I worked at local grocery store (Hy-Vee) as night shift manager. As soon as they found out I knew something about a computer I started updating prices, taking backups and running reports. I slowly slipped away from accounting and made it my minor and computer became my major.

Starting my junior year in college I began working remotely for a fim in Plano, TX that developed a client server system for machinery dealers. Just so happened my parents had upgraded to that system. I was responsible for support documentation. I had to reverse engineer a bunch of C code and author the various support manuals. This system was all PC based so I quickly learned about emulators, file and raw binary formats, Mac to PC communications and other such aspects.

After graduating college in 1991 I joined the developer division at State Farm. I was first assigned to the testing unit and responsible for all kinds of setup and issuance technologies. At that time State Farm used the HP All-Base system on regional mainframes. I had to write and maintain a ton of various processes allowing developers to properly test. I wrote in Cobol, Pascal and C depending on the project needs. I learned about testing harnesses and frameworks from this team. I developed a code coverage tool that would auto generate test plans. I wrote a JetForms to PCL conversion utility from the Mac to the PC that was used for mail merge activities. I wrote compression routines and much more.

While I was working for the testing team I was also developing on the Mac and PC at home on FoxPro. I was working with a friend (Doug Wright), building an application for a home decorating company that would combine their parts books and sales data. It seems pretty ironic now that when I joined Microsoft many of the original SQL Server support engineers came from the FoxPro team. Most of these folks are still part of the SQL Server support staff yet today.

While building the FoxPro application, I found I did not like the performance and taught myself how to use threads on the Mac. I actually wrote a basic, threaded database engine using Lightspeed C for the Mac that I eventually deployed. Looking back on my threading techniques today would probably scare me. I have learned so much at Microsoft about thread synchronization, memory barriers, spinlocks, bus invalidations and more.

In 1992 I moved to the CAT team and started the SQL Server adventure learning whatever I could about SQL Server and eventually the new NT platform. State Farm was moving towards a non-NT solution in 1994 so I as well as 3 other members of the CAT team joined the Microsoft support team in TX, and I have been here every since. The great thing is that I can help customers but I get to apply my knowledge. I still code utilities such as ReadTrace, OStress, SQLIOStress, SQLIOSim and others that make our support team more efficient.

SSC : How do you like living in Dallas?

Bob : I am actually located in Las Colinas, TX. In fact, this is the largest US SQL Server support site and has been so for many years. Since I could grasp what football was, I have been a Cowboys fan, so Dallas was the place for me.

SSC : Who's the most fun to work with at Microsoft?

Bob : There are a couple of engineers that I sit beside everyday that I would not go without. We have all been here 11 or more years and we work so well as a team. They have become my family. When it comes to the development team I am still partial to the engine team, but the entire SQL organization is great. However, the engine team has many members of the former SQL Server support staff as well as veterans. There are a bunch of core engine developers that have been with the product as long or almost as long as I have. I know them well, the communication is very easy and we think alike. It is nice to work with familiar folks that have the same passion to do the right thing as I have.

SSC : We've all heard stories of some characters at Microsoft. Any interesting ones that stunned you or surprised you when you first went to work in Redmond?

Bob : The great thing about Microsoft is that everyone gets to be themselves. When I worked at State Farm we had to wear a suit and tie. I grew up in Iowa, farm town with a graduating high school class of 12 students. I always expected a straitlaced approach to business. When I showed up at Microsoft if you have on a suit and tie you are considered interviewing. Folks with ponytails, wearing shorts, Microsoft providing free sodas, it was all pretty overwhelming.

I think the best story was my first day at work on Oct 1994. I arrived and was taken to my desk. Got some instructions on how to setup my PCs (plural only had one at any other job I now have 5 at my desk). A hour or so later my manager handed me a free t-shirt. (Little did I now that the free t-shirt's would continue to come a couple of times a year and I would not need to purchase any for work ever again). He then told me to that the team was going out for lunch and bowling and then to an offsite meeting. It just so happened the offsite meeting was a visit from SteveB. After a first day I had a new desk, with PCs, got a free lunch, went bowling and was treated to a SteveB meeting. I figured it was going to be a bit harder than that some days but this was a great start.

SSC : What's your current favorite tech gadget?

Bob : I have all kinds of things. PCs, handhelds, cell phone, video equipment, MP3 player, wireless equipment and the like. I have moved away from these a bit and into things like high tech depth finders and underwater cameras for the boat. The media PC is the latest thing that takes over our living room.

SSC : What's a high tech depth finder do?

Bob : It is a sonar system that allows you to view things underwater. The better ones on the market are often good enough to let you see the position of your lure relative to the fish.

SSC : What kind of boat do you have?

Bob : 18' AlumaCraft I love it as it is large enough to get up and go but small enough I can take it just about anywhere I want to.

SSC : Romo or Bledsoe? I'm a Cowboys fan as well, so what do you think of the Cowboys changes in 2007?

Bob : Drew is actually a really good guy. If you look at his foundation and the things he does in the community you can't go wrong with his as a person. I like Romo on the field because I like the way his passion comes across and he is such a natural athlete in so many areas outside of football. I love to work with people that have that passion to learn, do the right thing and get better at it each and every time they try.

SSC : What does Bob like to do when he's not working on SQL Server?

Bob : I love family time as I am a home body. I have a fishing boat so I get it out whenever possible. The kids and the dog love it on the water so the wife and I often pack up the cooler and tent and head for a lake weekend somewhere.

Bob Dorr

Caught Oct 2006 on Red River

Walleye 10lb, 32"

SSC :Where's the best fishing in Texas?

Bob : Texas has a lot of good lakes and reservoirs and is known for large bass and crappie. I grew up in Iowa so I am still partial to Walleye fishing but Texas is pretty warm for that. I go out to Lake Meredith, up to Texoma, across to Broken Bow, OK, Possum Kingdom is nice, Sam Rayburn, I will run over to Greer's Ferry in AR or up to Stockton, MI I try to get in a trip to Lake Erie every couple of years and to South Dakota yearly as well. If they are biting you will probably find my kids and I on the lake.