Server crash, less than 100MB of disk space

  • Yesterday a mail server nearly crashed due to major space issues. It turns out we had less than 100MB of disk space available, and the server is brand new.

    I have no way of accessing that server to see what reports are being run, what they're shrinking, or expanding, but:

    What the heck can I do to prevent this from happening again, theoretically? The recommendation was for everyone to shrink the size of their inboxes by purging the unnecessary, but I'm curious as to what others would do in this case.

    Thoughts?

    (Personally I'd be wondering how it got that far without some compression and examining the log files report.)

    _____________________________________________________________________
    -Jamie Scharbrough
    MCTS: SQL 2008R2

  • I'd get my shotgun and go find the clown who's in charge of the mail server. -Somebody- must be responsible for the machine - he should never have let it get to that stage. If you have no way of examining it, there's nothing you can do about it.

  • I like your enthusiasm!

    I was really ticked off when I heard about it, but because I'm a beginner, I usually keep my opinions to myself.

    I was also hesitant to post this, as I knew it was NOT good, but wasn't sure if I was missing something obvious somehow.

    Glad to see my instinct was right.

    I am curious though: theoretically, what would you do if your server reached that state?

    _____________________________________________________________________
    -Jamie Scharbrough
    MCTS: SQL 2008R2

  • Well, this isn't exactly SQL stuff, and I'm no expert, but...

    If I was in charge of the machine, I hope I'd have enough sense to keep it from ever getting into such a state, but if I was handed this as a problem to fix, because the original person in charge had absconded to Bermuda with the company's retirement fund:

    First thing I'd probably do is take the machine offline, to keep it from getting even further gummed up. If people bitch, I'd just tell them that I was handed a mess and I'm trying to fix it - if you don't like the way I'm doing it, call someone else. Nobody is going to die if they're cut off from their email for a few hours. Besides, if the machine's frozen, their email isn't working anyway.

    Second, look at what's eating all the space. There is an excellent bit of freeware called SequoiaView that gives a graphic view of an entire disk drive. It does wonders for getting a quick grip on your disk space. It's always one of the first things I install when I inherit a computer, and I have had it on every personal machine I've owned since I discovered it.

    At that point it becomes a bit of a detective game, depending on what's on the drive and what other hardware resources are available. If there is a big chunk of free space on a SAN or NAS handy, I might consider copying the entire drive there, just as a precaution.

    If there is a bunch of ballast in various temp folders, wipe those.

    Does the machine host an FTP server? If so, examine the the FTP-accessible folders. People have a tendency to store enormous amounts of trash there. Or it may have gotten hacked and some script kiddie has filled the machine with movies or porn.

    Does the mail server have a size limit on attachments? If not, it should. There is no theoretical limit (that I know of) on email attachment sizes, but almost every server has some sort of limit, precisely to keep things like this from happening. Some rookie can easily attach a folder instead of a file, and the attachment that was supposed to be a cute picture of a cat turns out to be the entire documents folder. Look at the individual email account folders - some braindead mail servers automatically return the attachment with the reply to an email containing one, so useless copies of things accumulate.

    If there is an auto-answer set on both ends of a conversation, the two servers can quickly build up a huge volume of mail constantly telling each other that someone is out of the office.

    Are there shares on the drive? If so, what's in them and who has access? Some people make backup copies of everything, with the thought that they'll make a copy now, to be safe, and sort it out later, when they have more time. Of course, that never happens and useless ballast piles up.

    Hope that gives you some ideas. There is not a point-by-point rulebook for such a situation; in every such case, you simply have to get into the machine and see what happened, but SequoiaView is probably your best starting point.

  • No, that was great! I hadn't heard of SequoaView; I was thinking of running reports and figuring out a) where the main space issue was coming from, b) where the wasted space was, and c) what could be compressed or sent else where (like a NAS or external).

    As for "No one's ever died from not having their emails for a few hours"...man, don't ever say that in front of lawyers. It is absolutely life and death to the ones I know if they can't check their email every six seconds. 😀

    _____________________________________________________________________
    -Jamie Scharbrough
    MCTS: SQL 2008R2

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply