June 15, 2018 at 7:59 am
I've worked in
Finance - care about testing to different degrees. Overall high. Usually a copy of production to test against
Education - Ha, testing?
Utilities - Some testing, but surprisingly poor in systems
Technology - all over the board
Construction - nah, it works on my machine.
I've only worked at a few companies in each area, so this is just anecdotal, not really evidence.
June 15, 2018 at 8:56 am
Jeff Moden - Thursday, June 14, 2018 9:16 PMEric M Russell - Thursday, June 14, 2018 10:35 AMA stored procedure can pass QA and still cause problems like critically poor performance in Production, and there really is no way to simulate a production environment. Sure, you can spin up a test VM with the same number of logical CPU and memory as production, but 2 sockets and 16 cores does not equal 8 sockets and 16 cores, the storage is totally different, and you're not testing within the context of a typical business day workload.To me, it seems dangerous to deploy a stored procedure to Production, without first having unit tested the new version in production. So long as you use a UnitTest schema to avoid altering any existing production objects and the procedure doesn't modify data, then the risk is minimal. It's essentially no different than running an ad-hoc query in production, except you're executing the code within a compiled stored procedure.
We actually do programmed load testing on a copy of the production database on the production server. We also run PEN testing both at the same time.
It sounds like you're talking about load testing an ETL process.
In my case, it's performance tuning stored procedures used by web services and reports, and I deploy them within the production database but to a UnitTest schema. It seems to me that unit testing against a copy of the production database on the production server would potentially be more disruptive to end users, because the database copy would consume extra disk storage, I/O, and memory.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
June 15, 2018 at 9:09 am
Eric M Russell - Friday, June 15, 2018 8:56 AMJeff Moden - Thursday, June 14, 2018 9:16 PMEric M Russell - Thursday, June 14, 2018 10:35 AMA stored procedure can pass QA and still cause problems like critically poor performance in Production, and there really is no way to simulate a production environment. Sure, you can spin up a test VM with the same number of logical CPU and memory as production, but 2 sockets and 16 cores does not equal 8 sockets and 16 cores, the storage is totally different, and you're not testing within the context of a typical business day workload.To me, it seems dangerous to deploy a stored procedure to Production, without first having unit tested the new version in production. So long as you use a UnitTest schema to avoid altering any existing production objects and the procedure doesn't modify data, then the risk is minimal. It's essentially no different than running an ad-hoc query in production, except you're executing the code within a compiled stored procedure.
We actually do programmed load testing on a copy of the production database on the production server. We also run PEN testing both at the same time.
It sounds like you're talking about load testing an ETL process.
In my case, it's performance tuning stored procedures used by web services and reports, and I deploy them within the production database but to a UnitTest schema. It seems to me that unit testing against a copy of the production database on the production server would potentially be more disruptive to end users, because the database copy would consume extra disk storage, I/O, and memory.
That and load testing the applications. We have software that simulates up to 1000 simultaneous connections (and some remarkable Developers that put it all together).
--Jeff Moden
Change is inevitable... Change for the better is not.
June 15, 2018 at 3:40 pm
The biggest problem with creating your own test data is – it is perfect data. Even if you try to introduce errors into the data they are mostly still within bounds or expectation.
Real data is the best way to actually prepare your product for the real world. I don’t think you could even put a room full of monkey on terminals to create test input, and still get the results from real data.
I worked on many projects that used "test data" and everything looked good, even great. But when we started using real data we saw anomalies we did not expect, much less anticipated.
And the problem with "Cleaning the Data" you may be actually removing the anomalies you need for testing.
I have no problem restricting the use of Real Data to "Only onsite testing" and only to higher level employees. Real data does need to be protected.
Especially mine!
June 15, 2018 at 9:18 pm
Frank W Fulton Jr - Friday, June 15, 2018 3:40 PMThe biggest problem with creating your own test data is – it is perfect data. Even if you try to introduce errors into the data they are mostly still within bounds or expectation.
Real data is the best way to actually prepare your product for the real world. I don’t think you could even put a room full of monkey on terminals to create test input, and still get the results from real data.
I worked on many projects that used "test data" and everything looked good, even great. But when we started using real data we saw anomalies we did not expect, much less anticipated.
And the problem with "Cleaning the Data" you may be actually removing the anomalies you need for testing.I have no problem restricting the use of Real Data to "Only onsite testing" and only to higher level employees. Real data does need to be protected.
Especially mine!
Heh... even with the idea that errors in data may be within the bounds of expectation, it's far better to do the testing with such data than to do none at all.
And, if you have a team that has seen the "Devils in the Data" before, I can assure you from both previous and current experience that there will be tests constructed that will make you ask, "How in the hell did they anticipate the faults of real life data so well?"
And, BTW... if you haven't thanked someone in QA today, you may be strongly remiss because they save your keester every day and you just might not know it yet. 😉
--Jeff Moden
Change is inevitable... Change for the better is not.
June 16, 2018 at 6:34 am
When I worked in the video game industry, our product was a massively multiplayer online game. On launch for example, we had around a million box sales from all over the world. Almost all products in this genre have significant issues on launch and even 6 months post launch.
Testing data does happen with a copy of production, but it also has to happen on production when pushed out to the hundreds of servers in the various data centers. I think we had something like 2 databases per game server with about 100 game servers across 4 data centers around the world.
Unfortunately, you cannot just simulate that and expect everything to go smoothly. Both the source code and data is too big. So much can happen. And when your product is a live one, where your consumers are burning before your eyes, you have to push out things super fast without having all the time for simulated tests.
Like connection testing for example. Can't really do that properly because you would have to simulate thousands of connections from all over the world connecting to one server with each connection doing normal things like walking, killing monsters, chatting, selling on market, etc. Really hard to replicate this to test data performance.
In another example, testing new bag storage for players and how the database handles bag items. Then not realizing you can put bags within bags (arrays within arrays) and realizing some jackass just stacked 100 bags within itself with items in each level. :crazy:
June 18, 2018 at 1:11 am
Frank W Fulton Jr - Friday, June 15, 2018 3:40 PMThe biggest problem with creating your own test data is – it is perfect data. Even if you try to introduce errors into the data they are mostly still within bounds or expectation.
Real data is the best way to actually prepare your product for the real world. I don’t think you could even put a room full of monkey on terminals to create test input, and still get the results from real data.
I worked on many projects that used "test data" and everything looked good, even great. But when we started using real data we saw anomalies we did not expect, much less anticipated.
And the problem with "Cleaning the Data" you may be actually removing the anomalies you need for testing.I have no problem restricting the use of Real Data to "Only onsite testing" and only to higher level employees. Real data does need to be protected.
Especially mine!
It shouldn't be perfect. It might be at the start, but you should add to it over time, as you discover cases that hit new boundaries. You ought to base this on the items that actually come up in production. The idea is just to have one case of each, not the thousands or millions of repeat cases in production.
If you only test the happy path, the expected things, you'll never catch many bugs, and you don't have good test set.
June 20, 2018 at 5:18 pm
Jeff & Steve,
Since the two of you seem to misunderstand me; and I have the greatest respect for you both. I must clarify.
I believe in Testing, Testing and More Testing. Test with everything you have and as often as you can.
I was just trying to say and I think it was Don Rumsfeld who said it best "Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know."
I stated test data is most often built from our expectations of the data we will get. The problem is our expectation are often wrong. Only the end user can come up with thing we don’t expect.
And yes as we find data that we didn’t expect we need to add it to our test set(s).
Thank you
June 20, 2018 at 5:51 pm
Frank W Fulton Jr - Wednesday, June 20, 2018 5:18 PMJeff & Steve,
Since the two of you seem to misunderstand me; and I have the greatest respect for you both. I must clarify.
I believe in Testing, Testing and More Testing. Test with everything you have and as often as you can.
I was just trying to say and I think it was Don Rumsfeld who said it best "Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know."I stated test data is most often built from our expectations of the data we will get. The problem is our expectation are often wrong. Only the end user can come up with thing we don’t expect.
And yes as we find data that we didn’t expect we need to add it to our test set(s).Thank you
Ah... my bad. I should have guessed that from the following comments you had in your post.
And the problem with "Cleaning the Data" you may be actually removing the anomalies you need for testing.
I have no problem restricting the use of Real Data to "Only onsite testing" and only to higher level employees. Real data does need to be protected.
Especially mine!
--Jeff Moden
Change is inevitable... Change for the better is not.
June 21, 2018 at 12:02 am
kerry_hood - Thursday, June 14, 2018 3:58 AMDoes anyone outside safety-critical industries (eg aerospace) actually work with proper test sets? Perhaps it's just my (30-year) experience, but I'm yet to work anywhere which has proper test data sets (that are not yesterdays/last weeks' backup restored), and that's across insurance, pensions, oil exploration, finance,merchant banking, and logistic sectors... everywhere wants their software yesterday and is not prepared to put in the time/money/people to test it properly.
Some 20 years ago I visited the workplace of my friend in Kiev, Ukraine.
He was working for a software company which was doing outsorce work for for a company named Apple Inc. You may heard of it, it's from America.
The company occupied 3 floors in an industrial building, can't tell how many employees, but surely it was a 3 digit numbers.
The company was hired to do one thing - testing.
Building test cases, writing test procedures, generating test data sets, producing reports.
They were not involved in the development/bug fixing at all. Just testing.
I know there was at least one another company like this in another Ukrainian city.
And I don't know how many of such companies were hired by Apple in other countries.
10 years later I was involved in developing an ETL / data warehouse system for a global fuel company.
Part of the requirements was to run a comprehensive test against the system prototype populated with 4 years worth (according to their estimations) data set.
Test was performed by a bot farm they've hired. Bots were launching typical web requests from about 50 different locations around the Globe, covering all inhabited continents, 400 bots from each location.
We had to disable DDOS attack protection to let the test go.
We passed the test when 97% of requests received expcted responds within 5 seconds. This time included network latency from, say, Capetown to Sydney.
Result - there were no slow response / sclability issues on the system after many years in production.
So, some companies actually do testing.
_____________
Code for TallyGenerator
June 21, 2018 at 3:06 am
Frank W Fulton Jr - Wednesday, June 20, 2018 5:18 PM...
I stated test data is most often built from our expectations of the data we will get. The problem is our expectation are often wrong. Only the end user can come up with thing we don’t expect.
And yes as we find data that we didn’t expect we need to add it to our test set(s).Thank you
I agree with this. However, I think this sidesteps the point. It's not that test data is flawed. It's always flawed because we have those unknown unknowns, but we should be working to minimize those over time.
Perhaps I didn't state this well that this isn't a set it and forget it. This isn't a point in time thing. This is an ongoing process.
June 21, 2018 at 3:13 am
Jeff Moden - Thursday, June 14, 2018 8:13 AMkerry_hood - Thursday, June 14, 2018 3:58 AMit seems that there are a whole lot of people that view testing (especially real QA/UAT) as some sort of annoyance ("expert" arrogance) on a daily basis. Really spooky out there.
It seems like this would be fairly easy to measure. How much time and money is going into QA and how much the release issues are costing.
That's assuming anyone is measuring, and those reports would reach influencers, and those people would be open to the idea / budget ... or even approve of the time to draw them up 🙂 On a day-to-day basis I am faced no one having time to test the changes users been screaming out for, understandably as the users have their own jobs to do, there's no BA, and only the developer to test (the DBA role here is a bit different to the norm as the main platform is an AS400 DB2 database running ~200 business-core RPG programs)... but there you go, the vagaries of being a developer DBA in-house.
June 21, 2018 at 7:40 am
robert.sterbal 56890 - Thursday, June 14, 2018 2:18 PMJeff Moden - Thursday, June 14, 2018 8:13 AMkerry_hood - Thursday, June 14, 2018 3:58 AMit seems that there are a whole lot of people that view testing (especially real QA/UAT) as some sort of annoyance ("expert" arrogance) on a daily basis. Really spooky out there.It seems like this would be fairly easy to measure. How much time and money is going into QA and how much the release issues are costing.
That's assuming anyone is measuring, and those reports would reach influencers, and those people would be open to the idea / budget ... or even approve of the time to draw them up 🙂 On a day-to-day basis I am faced no one having time to test the changes users been screaming out for, understandably as the users have their own jobs to do, there's no BA, and only the developer to test (the DBA role here is a bit different to the norm as the main platform is an AS400 DB2 database running ~200 business-core RPG programs)... but there you go, the vagaries of being a developer DBA in-house.
Maybe, but I rarely see anyone in these forums publish data about it. I'm not talking about a comprehensive analysis, just more openness on both sides about the challenges each other face. Probably something that gets better over time than resolved all at once.
412-977-3526 call/text
June 21, 2018 at 8:43 am
kerry_hood - Thursday, June 21, 2018 3:13 AMrobert.sterbal 56890 - Thursday, June 14, 2018 2:18 PMJeff Moden - Thursday, June 14, 2018 8:13 AMkerry_hood - Thursday, June 14, 2018 3:58 AMit seems that there are a whole lot of people that view testing (especially real QA/UAT) as some sort of annoyance ("expert" arrogance) on a daily basis. Really spooky out there.It seems like this would be fairly easy to measure. How much time and money is going into QA and how much the release issues are costing.
That's assuming anyone is measuring, and those reports would reach influencers, and those people would be open to the idea / budget ... or even approve of the time to draw them up 🙂 On a day-to-day basis I am faced no one having time to test the changes users been screaming out for, understandably as the users have their own jobs to do, there's no BA, and only the developer to test (the DBA role here is a bit different to the norm as the main platform is an AS400 DB2 database running ~200 business-core RPG programs)... but there you go, the vagaries of being a developer DBA in-house.
Maybe, but I rarely see anyone in these forums publish data about it. I'm not talking about a comprehensive analysis, just more openness on both sides about the challenges each other face. Probably something that gets better over time than resolved all at once.
All I have to offer is anecdotal evidence.
In a previous job, we started off with no QA Department and no peer reviews. The code was a train wreck of personal preferences with the main personal preference being to not include any comments (not even a flower box) and some people wrote their code as a single line of code. Deployments took forever because they almost always had to troubleshoot and repair some of the SQL that was being deployed. I was on one of the deployments because I had code that needed to be deployed by the DBA. I had one piece of code and another developer had 5 (it was the first time in my new position that I had code to be deployed). None of the other developer's 5 pieces of code actually passed even the "blue button" syntax check. It took six hours to deploy those 5 scripts because the developer refused to take part in any production deployments.
After looking at the code and doing a deeper dive on all the existing code, I went to the big guns to have a change made with all the reasons why.
It turns out that they were in the process of establishing a QA group (which actually occurred the week after that, thankfully). I also insisted that I peer review all SQL code and insisted on written standards and then handed them the standards (which also stated what the peer review process and other development/QA paths were for the whole SDLC including methods for getting or creating test data and a "Release Template" wrapper for all SQL code being released).
While all of that was ramping up, they did a release that was supposed to update just 1000 rows in the customer database. There was a fault in SQL Server (and MS made a hotfix for it) where an IS NULL criteria was ignored in the WHERE clause under certain conditions and it updated all 726,000 customer rows. The DBA had no clue as to what was expected and so committed the run. Next morning, it became apparent what had happened. That's also when they found out that the backups had actually been failing for weeks and no one was the wiser. The nightly snapshot to the reporting server had also run after the error occurred, so there was no easy way to get back to where we were. It took 40 people 10 days to undo the damage because all of the nightly runs had also used the bad data and I don't believe we ever got it back to 100%. The whole problem there was that there was no indication in the run results of the code as to what was actually expected.
Getting back to the path we ended up on... It took a while because there was a shedload of crap code that was uncommented and very difficult to read but the rule was, "If you have to touch it, you have to fix it". Of course, emergency releases for a production issue were the exception... the rule there was "You have to fix what you touched within the code but not necessarily all of the code). QA did the job correctly as did some of the managers and the better developers.
To make a longer story shorter, R&D for either a fix or for new/modified functionality on large stored procedures (some of them were 5,000 lines long) went from more than 2 days down to usually less than an hour. Kickbacks to Dev from QA for failed tests dropped from more than 80% (even on rework from previous rejections) to near zero because the Devs could spend time doing some quick unit tests (they were also taught how to quickly generate test data when not available through other means) and they also had time to "do it right" instead of taking 2 days just to figure out what needed to be changed.
Similar but slightly less sensational events have occurred at every job I've had except the current one, which has exceeded all of my expectations for all the same reasons. The Developers (both frontend and backend) take great pride in their code and brag about the killer work they now do and the performance improvements they make to legacy code. When I first started, the old crew (most of which are long gone now) did things like the previous company I spoke of originally did. There were multiple 10 to 20 minute blocking sessions each day, the 16 CPU server (largest database was only 65GB) was screaming at between 40 and 60% cpu usage, and batch jobs actually did need to run after hours because they also crushed the server for hours on end. Some wouldn't even finish in the 12 hour nightly time frame.
Today, some of our databases have exceed the Terabyte mark, there are many more databases than before, and our customer base has exploded. But we now enjoy an average of 8% CPU usage for "normal times" and we actually do run a whole lot of the batch jobs during the day because they no longer cause blocking and they now only take 30-60 minutes to handle 20 or 30 times more than what they used to.
In all cases, it was a painful process to setup the correct SDLC and to make the improvements to legacy code that were needed but, oh my... the rewards were fantastic. The duration of the full SDLC has dropped tremendously, there are only rare failures, and we've been able to handle much more scale and still beat the hell out of the original times when the original database were just a sliver of what they currently are.
Test data, unit testing, properly documented code, peer reviews and standards, time to QA/UAT, etc, all seem expensive and bureaucratic if you look only at those things but, if you look at the truly big picture, there's a wealth of monetary and time savings not to mention a huge increase in customer satisfaction that has caused business to boom because of referrals between similar companies as our customers.
--Jeff Moden
Change is inevitable... Change for the better is not.
Viewing 14 posts - 16 through 28 (of 28 total)
You must be logged in to reply to this topic. Login to reply