October 24, 2022 at 8:26 pm
Over the years of learning the hard way in database work, I grew to get a little better at good satisficing (perhaps manually importing some data short-term until it is automated) versus bad satisficing (granting too many permissions to "get it done" or poorly normalizing tables, etc.).
HTH.
-- webrunner
Another good way to look at the compromises we sometimes make.
October 24, 2022 at 8:58 pm
I'm concerned that after a year or so, perhaps sooner, management or someone in the fiscal office, will wonder why our cloud bill is so high.
They will. And there's nothing to be done if you don't invest in changing to work with the cloud. lift-and-shift will cost more over time. Though if it's opex, maybe they don't care. For Gov, however, they will.
October 25, 2022 at 3:18 am
At the risk of 'negative feedback' I'll add that after a couple years, I still get the error rejecting my first post effort with "Did you really mean to do that" making me send the same thing again. And now it even tells me that I've already sent that. So I check, and that is true. There's my comment. It happens both on my Android phone and my Windows laptop. But it's good enough.
Tsk, tsk. Come on now, Rick... it's "Good enough", right? 😀 😀 😀
EDIT: And then I see the last 4 words in your paragraph but I did "Good enough": 😀
--Jeff Moden
Change is inevitable... Change for the better is not.
October 25, 2022 at 3:27 am
What I'm amazed about is the attitude surrounding software projects especially from large companies like MicroSoft. What ever happened to the idea of "Putting your best foot forward" instead of publishing incremental junk? I think a large part of the problem is how easy it is to push a fix to the customer instead of just taking a little extra time to do it right the first time. It encourages the "good enough" attitude when it's really not right.
--Jeff Moden
Change is inevitable... Change for the better is not.
October 25, 2022 at 1:19 pm
What I'm amazed about is the attitude surrounding software projects especially from large companies like MicroSoft. What ever happened to the idea of "Putting your best foot forward" instead of publishing incremental junk? I think a large part of the problem is how easy it is to push a fix to the customer instead of just taking a little extra time to do it right the first time. It encourages the "good enough" attitude when it's really not right.
This is truly annoying. Was just working on an upgrade to our ETL platform and the vendor switched one of the components from an install as is from a downloaded to package to an install then run an update which has to pull from online resources. That's already annoying enough when installing to a server since it now has to deal with proxies and what not but there was a bug in the install that caused it to hang for some people. Which means you now have to upgrade that component leaving it out of sync with everything else so other things need to be updated as well.
Ship it broken and fix it later is I think a different issue than good enough.
October 25, 2022 at 1:41 pm
BWAAA-HAAA-HAAA!!! I have the "Change is inevitable" quote in my signature line below. I found its brother today quite by accident that covers what you went through, ZZ...
"If it's not broken, fix it until it is."
--Jeff Moden
Change is inevitable... Change for the better is not.
October 25, 2022 at 2:43 pm
Rod at work wrote:I'm concerned that after a year or so, perhaps sooner, management or someone in the fiscal office, will wonder why our cloud bill is so high.
They will. And there's nothing to be done if you don't invest in changing to work with the cloud. lift-and-shift will cost more over time. Though if it's opex, maybe they don't care. For Gov, however, they will.
I'll try to influence them to change their approach to software development. But sometimes I feel like I'm a tadpole that's trying to push an ocean liner to a new course.
Kindest Regards, Rod Connect with me on LinkedIn.
October 25, 2022 at 2:57 pm
"Good Enough", great song by Van Halen(Van Hagar to some 😉 )
I used to have some nightly batch jobs that ran for hours. I was tasked with trying to make the SQL more efficient. I was able to get it down to about a minute or two for each. Now that was 'Good Enough' as it saved several hours of processing each night. But I I kept trying to shave more time, in the end I spent way more time on it than any additional savings in time I made.
So as Jeff likes to say, 'It depends', when it comes to good enough. Yes my savings on these nightly jobs was good enough for the application. But if it was a web application, it wouldn't be good enough for 30 second to a minute response time.
-------------------------------------------------------------
we travel not to escape life but for life not to escape us
Don't fear failure, fear regret.
October 25, 2022 at 3:21 pm
Good enough can truely get you in a fix. I have an access database in the old .mdb version and an application on an old Win 7 machine that I have used since 2015 for some personal data. The application met my needs so I did not worry about keeping it up to date. Now 8 years later, I want to retire the old machine. In upgrading my MSFT Office suite I find that this application no longer works with the new Access. The first issue is that I have archived 14 versions of the application that I never installed. The latest version of the application errors out trying to use the existing Access mdb. Somewhere in the 14 versions the database has been altered but I have no idea where, and can find no online info on matching the Access db to versions of the application. The application developer has long since ceased operations, so there is no help. This is 'good enough' for a time-consuming project for this winter.
Rick
Disaster Recovery = Backup ( Backup ( Your Backup ) )
October 25, 2022 at 6:58 pm
glhohman wrote:Excellent article, ...
I absolutely love it when the challenge at hand requires a graceful and carefully engineered solution, but I also find it gratify to simply craft something that works. In my mind I still see the perfect, but my heart is gladdened to see peoples lives improved by the small thing I did.
Thanks, and a great sentiment there. The value to the customer is king.
Heh... I absolutely agree with that last statement. That's also why I say that "Good Enough" frequently isn't. The people in the shop think it is... the customer doesn't. People also forget that, sometimes, the customer is internal... Might even be the DBA. If your junk is killing the server then, ultimately, you've disappointed your customer.
--Jeff Moden
Change is inevitable... Change for the better is not.
October 26, 2022 at 6:00 pm
Often a project has multiple customers and multiple perspectives. A developer trying to satisfy an external customer, might prioritize that party over an internal customers like a DBA. Time is money, after all. It's hard to say no to your boss when he says "ship it".
October 26, 2022 at 7:00 pm
Often a project has multiple customers and multiple perspectives. A developer trying to satisfy an external customer, might prioritize that party over an internal customers like a DBA. Time is money, after all. It's hard to say no to your boss when he says "ship it".
Absolutely true...
One of the things that I'm trying to get at is that a little time spent up front can save some huge embarrassments because that time up front can and should actually define both what is "Good enough" and how long it should take. That's not what usually happens, though. Too often, people, like DBAs aren't even invited into the conversation even on major projects... and almost all such projects involve a database.
--Jeff Moden
Change is inevitable... Change for the better is not.
October 26, 2022 at 9:50 pm
"Ship it broken and fix it later is I think a different issue than good enough."
I totally agree and I can do you one better, although I don't think of this dialog as a contest, "Ship it broken and refuse to fix it". I think I can tie this story back to “good enough” as well.
---
I was working as an IT manager at a medium sized Anatomic Pathology Consulting Firm who, just before I came on board, had started a Clinical Laboratory. Now this Consultancy had promised their clients 4 hour turn around on common tests. Things like Microbiology or Flow Cytometry take a lot more time, but for simple blood work this was do-able. The problem we had, one which I was told I needed to fix, was last mile one.
Specifically, the problem we had was with the delivery of faxes for the Clinical Lab. The vendor whose ordering and resulting systems we were using had specified that the fax server, the only fax server software they supported, was Zetafax by Equisys. As we grew in volume, and this growth was happening very quickly, we were dealing with more and more faxes that needed to go out. We had told our providers that we would turn around results in four hours, from the time our currier picked up the specimen to the time they had their result, but here’s the rub. 95% of the providers in question receive their results via Fax, if you can't deliver a fax in a timely fashion you simply have not delivered.
What we saw was that when result volumes went up, the fax server would simply freeze. No notifications. No indication that there is a problem. Providers started calling in saying “I haven't received the results for so and so”. Our client services people would look it up in the source system, the system that was sending the faxes to Zetafax and it would say delivered. That was a misnomer on the part of that vendor since the system had no idea whether the fax was actually delivered or not. What it should have said was “Files Dropped”. That system dropped off a TIFF file and a text file with the extension CTL. Zetafax was then to pick up the CTL file, read it, and then send the associated TIFF file as the CTL file indicated. These freeze-ups were becoming a major problem for us.
We couldn't recreate these failures in TEST, so eventually I went to my CEO, and the rest of the organizational leadership, and told them “I’ve to do something in production. I have to run some debugging tools, and there's going to be an impact, I don't know how long the impact's going to be, but we have to do it if we are to figure out what is going on. They all said OK, we've told you that you need to fix this ASAP, so with all proper scheduling I shut down the fax server installed Procmon from the sysinternals suite. Within 15 minutes of starting the two required user mode processes we saw what the problem was. The Zetafax Client, and Zetafax Server, both user mode so must be started at the System Console or RDP Session 0, had deadlocked on the same CPL file.
I was fortunate enough to have an excellent network administrator working for me that not only understood networks and clients and servers, but also software development and had himself written multithreaded code in the past. I’m pretty sure the Zetafax System by Equisys had been a 16-bit Windows program, and when they ported it to Windows NT, they must have just recompiled it as 32-bit without taking advantage of the features of the platform like pre-emptive multitasking and threading.
My netadmin and I brainstormed this for a while, but neither of us could come up with a solution that didn’t involve the vendor re-engineering. When we contact Equisys and told them of our discovery, they flat out said that there were no plans to re-engineer the product. When we went to the Order and Results system vendor and asked if they could work with another fax server solution, the answer from them was that they didn’t have the resources.
When we brought our findings, and both vendors answers’ to our IT Steering Committee, to say they were shocked would be an understatement. My CEO just stared at me, and asked in quiet, calm voice, “Greg, what can we do?” The room erupted in a cacophony of voices, all the while my CEO continue to hold my gaze. Ideas were thrown out like “We could hand fax everything” or “we could send reports by Courier”. All these suggestions were shot down quickly.
When the room finally became silent, I said to our CEO, “the only thing that I can think of doing is replace Zetafax with a system we know works. We already have Facsys, the product that does fax delivery for our anatomic system, and it is rock solid. We know how the upstream system drops files for Zetafax to pick up. We could write our own software that picks up those files and delivers them to Facsys. I've already investigated, and Facsys has an API available. Using the Facsys API would allow us to not only send faxes, but also monitor success and failure. “This will be a significant capital expenditure”, I continued, and began to list off the things that we would need, including some of her people’s time to help me define the administrative interface. My CEO then stopped me and simply asked, “can you get me an estimate by 5:00 o'clock today? The board is meeting this evening”. I told her I could, and I did. The next morning, she told us “the capital was approved by the board, and you can work with our quality control supervisor to design the interface”. She went on to tell me “This is your highest priority”. We were off to the races.
Thankfully I had already crossed trained my netadmin on all of the work that I did on a daily basis: server work, system monitoring, and maintaining electronic interfaces. Since the mission was to get this new solution up and running ASAP, I immediately started in on making the purchases that were needed, getting wiring and telecom resources ordered, and working with our quality control supervisor to define the front end.
The whole system was built using C# and Winforms for the front end, C# and a couple of multi-threaded Winforms apps for the server side, with SQL Server as the backing database. When we had tested everything thoroughly, including a live demo for all the people who would be using the product, we started planning for the release.
I went to my netadmin and told him that my intention had always been to write this as a Windows System Service so that it could run unattended, but as it stood currently, I had written two user mode programs just like Zetafax. Each of the forms had exactly two buttons, one that said “OnStart” and the other “OnStop”. These are, of course, the Entry and Exit calls that a Service uses. I felt it would be easiest to go live with these programs running in debug mode, with breakpoints set pretty much everywhere a “Try” failed over into a “Catch”.
Here is where the good enough part comes in. When I showed my netadmin the two server-side programs, I told him that my plan had been to make these Windows Services, he just looked at me and said, “well it's no different than what we have to do right now with Zetafax”. I just said “Yep”, and that was that. When we went live, we’d already done all the testing we could do in our TEST environment, and could only see what happened under load by doing it in PROD. I quickly found a couple glitches, and I was able to fix them in the TEST environment have people test it in the TEST environment, and quickly roll the changes out to PROD so we didn't have to do a rollback. Only twice did we witness a failure, and by noon of that day, I think, we were live on the new system of our own design.
This was a system that I was very proud of because it really made a difference in people's lives, it saved the business’ reputation, and it met the quality expectations of a very excellent organization. I never did re-write it as a system service. It was Good Enough as it was.
October 27, 2022 at 4:01 pm
Thanks for the story glhohman
October 31, 2022 at 12:02 am
Thank you Jo for the notice.
It was a good exercise to write it up again. I have a version that I wrote up for a different audience, but I think you folks are unique and deserved your own version.
It has been very rare for me to write publicly. It's not like I'm shy or anything, I've talked to crowds over a hundred, it's just that in the past I've tried to stay under the radar. Now, I feel the need to start my own Consultancy and that means going public. Strangely enough, the only web communications on technical subjects I had done was with the Internet Storm Center. I used to visit there daily, and on a couple of occasions felt the need to contact the Handler on Duty to find out if there had been other reports.
Today, I see the Handler on Duty posted about an update to procmon and sysmon, the very tools I used to discover the problem with Zetafax. Convenient.
Viewing 15 posts - 16 through 29 (of 29 total)
You must be logged in to reply to this topic. Login to reply