November 13, 2018 at 6:30 am
I'm not sure if this is the right forum to post this but:
Our production server has been having a problem with these latest Windows Updates:
KB4462949 (10/13/2018)
KB4462923 (10/13/2018)
KB4462915 (10/13/2018)
KB4462927 (10/25/2018)
Our users could not connect to SQL via their applications after these updates were installed and OS reboot. Attached is an example of the error the users were getting.
Rolling back the Windows Updates and rebooting the server cleared the connection issue. However, now we have 2 high vulnerabilities in our Security scans.
These same updates were applied to other Windows servers that host SQL Server, and there was no issues there.
When users couldn't connect to SQL, I checked everything, Event Logs(Windows/Security/Application), SQL Error Logs and nothing was there. I checked with other Admin to see if a port was being blocked, or if HBSS caused the issue, and nothing lead me to any reason why these updates failed, and cause a disconnect between SQL Server and the applications in
production.
All SQL Services were running, and nothing in the logs even mentioned a problem with SQL. I tried rolling back each update one by one to see which one was caused the SQL disconnect, and they caused the same problem. I ran a health check on the server and that too didn't show anything that would help. So this is the issue I'm having. If you have any advice, I'd very much appreciate it.
Thanks
November 13, 2018 at 7:59 am
I'm surprised that any of those KB's caused you to lose connection to the SQL Server. Did the same issue happen when deployed these KB's onto your test environment (I assume not, as your now deploying on Production). If it did, then have you replicated the steps you took to fix that?
Did you validate the the SQL Server instance was still running? If not, what did the logs say when you tried to start it (and it failed). Was it just "MRS" (not sure what that is) that couldn't connect? Could other applications like SSMS connect fine? If it couldn't what was the error in SSMS (As that might give a better error).
If the instance was running, could you see the connection attempts from MRS and SSMS in the server's logs? If yes, what did the failure messages say?
Thom~
Excuse my typos and sometimes awful grammar. My fingers work faster than my brain does.
Larnu.uk
November 13, 2018 at 11:53 am
Thom A - Tuesday, November 13, 2018 7:59 AMI'm surprised that any of those KB's caused you to lose connection to the SQL Server. Did the same issue happen when deployed these KB's onto your test environment (I assume not, as your now deploying on Production). If it did, then have you replicated the steps you took to fix that?Unfortunately I cannot test these KB accurately on mytest server since the applications like "MRS" are not installed onthe test server, and the "MRS" database isn't on that server either. BTW, MRS is an application that Radiology uses to take mammograms and store data. The database is encrypted on my production server, and itis a long painful struggle just to get that database on my production server.
Did you validate the the SQL Server instance was still running? If not, what did the logs say when you tried to start it (and it failed). Was it just "MRS" (not sure what that is) that couldn't connect? Could other applications like SSMS connect fine? If it couldn't what was the error in SSMS (As that might give a better error).
Yes, SQL Server instance was still running, however theSQL Agent stopped. I restarted it, and SQL Server Agent was running again, butusers still kept getting that SQL error in MRS. I check all logs, even the logs on the users workstation,and the only "failure" message I saw was the user trying to log inand the login failed because there was no SQL connection. SSMS connected fine. In fact, and here is what's reallyweird, when I went into Activity Monitor in SSMS, I can see the usersworkstations logged into SQL Server?? (see attached). And the error logs in SSMS show only a failed login for the user. Which makes no sense because the user's permissions didn't change in the Security folder for that database.
If the instance was running, could you see the connection attempts from MRS and SSMS in the server's logs? If yes, what did the failure messages say?
Same as above.
November 14, 2018 at 8:39 am
Actually I was able to pull an SQL Log from that day. There seems to be a big gap between from the time the Windows Updates were installed and the OS rebooted (around 3am), till the time they were removed (around 11am). The user didn't report not being able to connect to SQL around 7am. Also, the Windows Event Viewer logs still showed no mention of SQL connection failures or Windows Updates issues for that day and times.
Not sure how to further troubleshoot, but any advice is appreciated. Thanks.
November 14, 2018 at 9:27 am
What I would do is have other apps that can connect to the instance (SQLCMD, SSMS, etc.) and verify they work after the kbs are installed. Ultimately, without a test environment, you'll have to retry on prod. BEFORE you do that.
You also could set up a copy of the MRS software and point it to your dev server. Create a database on that server that has the same name as prod, apply the updates , and try to connect. You'll likely get some errors because tables don't exist, but that's fine. You want to verify connectivity, not application functions. If this works, likely you have a different issue. In that case, you may need to apply the updates when you have downtime and start digging into the issues.
November 14, 2018 at 11:51 am
Steve Jones - SSC Editor - Wednesday, November 14, 2018 9:27 AMWhat I would do is have other apps that can connect to the instance (SQLCMD, SSMS, etc.) and verify they work after the kbs are installed. Ultimately, without a test environment, you'll have to retry on prod. BEFORE you do that.You also could set up a copy of the MRS software and point it to your dev server. Create a database on that server that has the same name as prod, apply the updates , and try to connect. You'll likely get some errors because tables don't exist, but that's fine. You want to verify connectivity, not application functions. If this works, likely you have a different issue. In that case, you may need to apply the updates when you have downtime and start digging into the issues.
I ran SQLCMD on production and got a SQL connection error. Named Pipes are enabled so something is a miss. I'll try setting up the MRS software on my test server and see if that connects. Thing is, one user is using has Windows 7 on their workstation, and another using Windows 10 on theirs. So operating systems, receiving different updates and different policies could be a factor. Also, it was just MRS, but several other applications that connect to that server.
There's also the issue of KB3177467 being applied after the other updates instead of before:
https://support.microsoft.com/en-us/help/4462923/windows-7-update-kb4462923
https://support.microsoft.com/en-us/help/4462927/windows-7-update-kb4462927
https://support.microsoft.com/en-us/help/4462915/windows-7-update-kb4462915
.
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply