June 26, 2024 at 12:00 am
Comments posted to this topic are about the item Where to Test Your Code
June 26, 2024 at 10:16 am
I do a lot of Python development for cloud environments. When designing and writing software I ask myself "How can this code be tested robustly in a mechanical way"? Often at least as much thinking goes into the test approach, especially for cloud integration tests. With it being the cloud and inherently a distributed system it can be difficult to test even with careful thought.
With AWS there is a library called Moto (the main AWS client library is called boto). This does let us create testing mocks so we can test much locally. I haven't come across anything similar for Azure or GCP so if anyone has experience with this please shout up.
I find unit testing is great for spotting embarrassing errors in individual functions. The way I think of unit testing and integration testing is
I don't mock the DB as such. As it is possible to create Docker images for many DB platforms, that is what I do. For things like Snowflake and BigQuery the Docker image approach isn't possible.
I would caution against rejecting DB testing on the grounds that you can't simulate production loads or volumes. Those things are obviously important but being able to test that something mechanically works, albeit at lower volumes/load, is a big step forward. I have seen integration tests pick up surprising bugs in the way the app uses the DB. I must reiterate that there is a lot of careful thought that needs to go into designing and building your tests and everything that goes into them.
Some aspects of DB testing are not those that have widely available solutions.
June 26, 2024 at 12:16 pm
Typo: bit
June 26, 2024 at 2:33 pm
Learning how to do unit tests is what got me my current job. I never did unit testing in my previous job, so I picked up unit testing and am glad I did.
However, when I entered my current position 9 years ago, they hadn't done any unit testing, either. So, we learned together, which was a growing experience. The first thing we tried to do was 100% code coverage. But that turned out to be crazy, as it was silly to test to see if assigning a value to a property of an object, would return the same value. (No, threading involved, so it really wouldn't change, between assignment and retrieval of the value.) So, after spending many weeks to months just verifying that
x = y;
always returned the same value for x, we gave up on the idea of 100% code coverage.
The one thing I didn't agree with was when my former boss insisted that all we should do is write integration tests, and he'd call them unit tests. Ultimately, the only thing that mattered to him was that the value that came out of the database was what was expected. But when the "unit test" failed, then it was always an issue of trying to figure out if it failed in the unit test, the code the unit test was testing, the network, or the database. I prefer writing both unit tests and integration tests, but that's not what we were supposed to do. Oh well.
Kindest Regards, Rod Connect with me on LinkedIn.
June 26, 2024 at 3:00 pm
Learning how to do unit tests is what got me my current job. I never did unit testing in my previous job, so I picked up unit testing and am glad I did.
....
Any testing is good, though it's some level of debt to support, so these need to be trimmed and managed. I think the trick of testing early is to understand you'll do silly things like check x=y, or that insert into x (y) value (z) means that z is in the table. In general, we shouldn't test things that are handled by the platform.
Part of testing maturity is learning what tests to write and not to write. I think that testing should often focus on bugs, or things your team does wrong (or repeats because of copy paste).
Also, I think 100% coverage is silly.
June 26, 2024 at 5:08 pm
Test coverage as a metric is a mine field. It's like page life expectancy or buffer cache hit ratio. It's all in the context in which they are used and in what combination.
A few years back I had to upgrade a number of Python applications where there were no tests to help evaluate whether changes would cause breakages or not.
The code was huge and convoluted and when I added a test for a bit I felt I understood I was shocked to get a very high test coverage percentage. The metric was based on the number of lines touched by tests. As the code was a sprawling mass of interdependencies a lot of lines were touched by a simple function call.
Test coverage as a metric needs to be considered when combined with the metric of cyclomatic code complexity.
These metrics only give value when they are used with honest intent and knowledge of how they should be used. They should be a product of behaviour rather than a goal in their own right
June 27, 2024 at 12:56 am
Test coverage as a metric is a mine field. It's like page life expectancy or buffer cache hit ratio. It's all in the context in which they are used and in what combination.
A few years back I had to upgrade a number of Python applications where there were no tests to help evaluate whether changes would cause breakages or not.
The code was huge and convoluted and when I added a test for a bit I felt I understood I was shocked to get a very high test coverage percentage. The metric was based on the number of lines touched by tests. As the code was a sprawling mass of interdependencies a lot of lines were touched by a simple function call.
Test coverage as a metric needs to be considered when combined with the metric of cyclomatic code complexity.
These metrics only give value when they are used with honest intent and knowledge of how they should be used. They should be a product of behaviour rather than a goal in their own right
Very good points, David. I look back at those times with the realization that we were only beginners at writing unit tests. Trying to do 100% code coverage seemed the right thing to do, but we'd never done any unit testing before. Baby steps. Well, we learned better.
Kindest Regards, Rod Connect with me on LinkedIn.
July 16, 2024 at 5:44 pm
Unit tests do not require production size databases, although load tests do and are important as well, once all the unit and integration tests are complete.
Code coverage is important, but as @david.Poole pointed out, how that metric gets calculated is important. The goal is to ensure that some percentage of the code paths are exercised (100% is a great goal, but 85% is probably more practical). Whether a path is exercised should be a binary (true/false) without considering how many times it may have been exercised. There are great tools for this for C#. For TSQL, not so much.
July 17, 2024 at 1:57 pm
Unit tests do not require production size databases, although load tests do and are important as well, once all the unit and integration tests are complete.
Code coverage is important, but as @david.Poole pointed out, how that metric gets calculated is important. The goal is to ensure that some percentage of the code paths are exercised (100% is a great goal, but 85% is probably more practical). Whether a path is exercised should be a binary (true/false) without considering how many times it may have been exercised. There are great tools for this for C#. For TSQL, not so much.
I agree that unit tests shouldn't require production sized databases, but not everyone does agree with that. When we first started doing unit tests my boss insisted that we really do integration tests, using the unit tests framework. And we had to use our test databases for the backend, which in our case was always a recent copy of the production database.
Kindest Regards, Rod Connect with me on LinkedIn.
July 17, 2024 at 3:58 pm
When we first started doing unit tests my boss insisted that we really do integration tests, using the unit tests framework. And we had to use our test databases for the backend, which in our case was always a recent copy of the production database.
The win here is the boss insisting on any form of testing. The battle between the different types of test would be one for the future.
Unit tests should be small, fast and tightly focused on specific behaviours.
Integration tests are broader and inevitably less fast.
performance and stress tests can be difficult to set up and slow to execute by their nature.
There is a whole load of human traits that drive what is considered an acceptable test approach. Corporate and personal attitude to risk being one.
I've learned that it helps to measure progress in terms of whether today is an improvement on yesterday. Then plan for tomorrow
July 18, 2024 at 4:19 am
I guess our code coverage is 500% plus a post prod kicker...
DEV
QA
DEVREVIEW (DBA Review and regression checks)
STAGE (UAT)
PREPROD (UAT and full system checks on full size from nightly restores which kills a flock of birds with one stone)
PROD (with immediate smoke testing after deployment)
As a result, we have nearly zero production glitches in the last almost 13 years and the ones we had were either simple enough to fix that night or the very next day.
--Jeff Moden
Change is inevitable... Change for the better is not.
July 18, 2024 at 1:44 pm
That is really awesome, Jeff! In fact, that's outstanding. In my environment we do unit testing and UAT, and that's about it. We don't have the success rate that you've mentioned you have. We definitely could improve.
Kindest Regards, Rod Connect with me on LinkedIn.
July 19, 2024 at 3:38 am
That is really awesome, Jeff! In fact, that's outstanding. In my environment we do unit testing and UAT, and that's about it. We don't have the success rate that you've mentioned you have. We definitely could improve.
Thanks for the feedback, Rod. I'm very fortunate that we have a management team that agrees... don't use our Clients for QA... instead, DO IT RIGHT THE FIRST TIME.
Our bottom line, customer loyalty, and new business is proof that it's absolutely worth it. 😉
--Jeff Moden
Change is inevitable... Change for the better is not.
Viewing 15 posts - 1 through 15 (of 16 total)
You must be logged in to reply to this topic. Login to reply