Best practices - How an app. should behaves when the DB server is gone

  • Hello Everyone,

    I would like to know how an application should behave (best practices) when the DB server is not availlable (because of a database crash or any other reason including maintenance)?

    Also, how do your own apps behave?

    We have a debate here on how our application should behave when this happens. Some people think that an application crash is ok while others think that we should display a message to the end user and ultimetly allow them to resume work when the DB server is available once again.

    Best regards,

    Carl

     

  • I'd like to think it should degrade gracefully and put up some message that things are not available now.

  • Thanks Steve and what about the possibility to resume what was began before the "crash"?

     

  • If every recordset is somehow presented on the forms, then the operations should be able to resume... but I can see a lot of times where that scenario could fail...

  • I fully agree to Ninja's remark. It is not worthwhile to design and program an application which can recover from every possible failure of one of it's components. A system has to fulfill a business purpose in the first place. If a server is critical to a certain business process you might have to invest more effort on the back end side.

    Better make sure that the user get's a proper message what the trouble is and that your LUWs (logical unit of works) are designed in such a way that, if a server breaks down, your data integrity is not corrupted.

    Cheers

    Rene

  • I think this comes down to the type of user and the units of work involved. Let's assume we're talking about physical users (not service end points) in which case how much effort in involved in recreating a unit of work in the event of the server being unavailable and similarly is the client application any use at all without the server. These types of applications have being recently dubbed "smart clients", by which it means the client app can be used while disconnected from the server e.g. Outlook.

    So I guess the question is will the user want to be able to do things while disconnected and is it worth the development effort to "resynchronise" when the server comes back online ?

  • Ultimately, the application should be able to resume.  If you wanted hibernation support for an application, this would be necessary as hardware such as wireless network connections will always leave you with the chance that your application would lose communication for some time.  So, from the Microsoft standpoint, a temporary disconnect should at least be supported to handle standby and hibernation situations.

    On the other hand, supporting this from a programming standpoint can be really complicated as it can involve holding state information at the client and server end and reestablishing connections when things are back to normal.  In corporate IT situations, this is probably not so necessary and spending money and time on it is probably not worth the effort.

    I would suggest that it is not good practice to have unhandled errors so an application that loses a necessary connection should at least give a useful error message and shut down gracefully.  It should also give some kind of useful message if the user starts it before the database is available again.  If you get a generic application exception, crash windows, or have to close your application with task manager, I would have to say it could use some work.

  • I think most, if not all, SSC members would agree that you should handle such an error gracefully, and let the user know what happened.

    As far as being able to resume where the error occurred without re-entering data, it would depend on the type of application, web or desktop, and the overall architecture. IF a desktop app, particularly in .NET, I would think you could design it to save information in XML files and then re-load them when the application is restarted. This could also be presented to the user an option after the error is detected. As Michael Earl said earlier, this would be complicated programming, but programming is supposed to make the user's life easier, not the programmers, and most programmers I know welcome challenges like this.

  • Use caution about storing state in unencrypted files of any format. One problem is that you may be leaving identity or other valuable information just sitting on disk (think of an order typed in that you can't save at the moment), but it also leaves a potential backdoor for an enterprising user to modify the file on disk to get around rules enforced in the UI. Consider handling this type of thing on the middle tier rather on the client.

    I agree with most of what has been said already. Not all apps need to be bulletproof, just bullet resistant! The worst apps are the ones where the connection fails, then you get about 25 error messages to click through because the connection failed, all the downstream objects arent there or are wrong.

    Beginning programmers using a client server model will just open a connection and leave it open, thinking that if it opens in the beginning all is well. Of course things can happen later, but more mature applications will open a connection, do something, close the connection so that pooling works. Getting a good centralized error handler is key, but not as easy as it sounds if you want to be able to recover when the server goes online.

    Finally, I'll add that just all db errors should be handled well, not just the connection. The two most common offenders are command timeouts and deadlocks.

  • All you have to do is setup a 'status' table in the database that shows what state the database is in. Then just have the application check that table to see if the database is down or not.


    Live to Throw
    Throw to Live
    Will Summers

  • Good idea! Never thought of it. I will tell my boss...

    I'll wait for you to implement it first and tell me how it goes...

    Carl

  • I agree with Andy about saving information, but when I made that suggestion I was thinking about places where I have worked, and, to be honest, most users, in my limited experience, do not have the time, nor the savvy to find saved information and change it without using the application. It has always been developers you have to worry about doing something like that!

    Also, nothing is more frustrating to a a user, and remember we all are users at some level, is an application that does not gracefully handle easily identifiable errors like the one we are discussing and the ones Andy mentions. In one company I worked at there was and probably still is an application that will present the user with the error and when you acknowledge the error message the application exits! The programmer caught the error, but still allowed the application to crash!!

  • I already have the table in place and the code is in production. We didn't need to test. I'm sure it will work.

    Seriously, this problem will depend on the type of app that you have. If it's a web app and your server goes down, then unless you have the webserver and database server on different machines there's not much that can be done unless you have a backup machine. If you are running a client-server app, then the application should already have a 'soft landing' for when it cannot connect.


    Live to Throw
    Throw to Live
    Will Summers

  • We didn't need to test it, I am sure it will work. Wow, that is a statement I don't think any developer should make.

    Also, if the DB is down a status table in the DB does not help you because you will get an error when querying that table. If you do as Andy has suggested and close connections when done with them, then the DB could go down between querying the status table and your next action.

  • The status idea is useful for times when you (the dba) want to force an app offline. You may need to a repair or quick schema change and need everyone out. Having a switch you can flip is really nice compared to killing spids and trying to flip the db to dbo only. If you really want to run with this idea combine with your application version, so that you can shut down old versions that should have been upgraded but havent. This lets you know when its same to make changes that would break old versions.

Viewing 15 posts - 1 through 15 (of 29 total)

You must be logged in to reply to this topic. Login to reply