It's done.

The big move is complete, from the old collocation site, really a spare room with a T-1, into a real live, collocation facility located in the Denver Technical Center at Viawest. Since it was a little more of an effort than I expected, and because Andy and Brian wanted to get the details, I decided to take a few minutes and jot down what I remember from the move. If I can get my digital camera working (the card got fried somehow), I'll take a few pictures.

Friday, 9:00pm

Since the web site will be down for 4-5 hours, I decide to stage a maintenance page. So I create one, be sure to copy the logo to the mail server, setup a new web site in IIS and test the basic page. It's simple, a "we're down, come back later" page. I go to start the web site and ..."

The web site could not be started and some other process controller or something else error.

Like an idiot I try to restart it 2 more times with the same result. After I stare at the screen for a few minutes I decide it's not worth it and get on with the rest of my Friday night.

Saturday, 8:00m

I have the brilliant idea while getting the kids setup in front of the TV. I'm about to make coffee when it hits me. Shut down the list server (which has it's own web server) and then the maintenance page should work.

So I run upstairs, shut down the mail software, and IIS starts my web site. I'd added it to a 2nd IP, so I test it and it looks fine. Now to repoint www.sqlservercentral.com over here. I log into the DNS zone provider we use (ZonEdit), well I attempt to. The password doesn't appear to be working. So I request a new password.

Which goes to Brian.

And he's not home.

What's that Murphy thing again? Still it's not the end of the world as I'd expected for the site to go dark anyway. I had hoped to be able to provide some status information, but as I said, this isn't the end of the world and as it turns out, I wouldn't be updating things anyway.

You might wonder why we have Brian setup for the service and not a generic account. Well when we started down this road, everything was on a tight budget and we usually did things ourselves as needed and then got reimbursed. Or not. But we haven't moved many things because it's a hassle and doesn't provide and benefits. This is one of the rare times that I couldn't get ahold of Brian to have him reset the password and I wasn't going to worry.

I grab the previous night's backup and copy to the web server and close down as the kids are calling me.

Saturday, 10:00am

I wanted to be walking out the door by now. No coffee yet as I woke up late and the little kids (3 and 5) are keeping me busy. My wife takes care of the neighbor's horses when he's out of town, so she's gone taking care of that. And I forgot my 12 year old has roller hockey, so I have to kick him out of bed, get him to get his gear in the car. Fortunately my wife arrived as we were walking out the door, so I got out the door by 10:30. After a detour to drop off my son at hockey, I'm on my way by 10:45.

I have a nice drive up to meet a friend, who is also my network guy and runs the firewall and general networking for SQLServerCentral.com. I arrive at his house to go and get servers, but first a delay.

"I need a new battery for my smoke detector. Come with me to get one"

Now normally I'm not really interested in hanging out when I'm about to take major downtime, but he's a friend and he's selling his house, so he doesn't want things not working. And nothing's down yet, so I agree. After a nice 20 minute detour to get that squared away, it's off to the servers.

Now this part goes smooth. At least at first.

Saturday, 12:30pm

Server shutdown time. Since we run headless most of the time, there isn't a monitor attached to the two servers that power the site. So I grabbed a spare monitor and plugged it into the first server I came to. Hitting "Enter" a few times brought up the login screen, so I logged in and powered down the system. So far things are going well. While we're doing this, we're taking everything out, so we power down the switch we'll be moving and unrack it along with some spare cables and parts we have laying around. I unplug the monitor and move to server 2 and plug it in. I wiggle the keyboard and ...

Nothing.

These are DELL 2650's and have video and keyboard on the front and back, so I go around to the back of the rack, plug in and...

Nothing.

I remember this. This was our first server, well actually the 2nd, but the first was a desktop pressed into server service. This was our first real server, one we purchased as soon as we could afford it, with dual 1.4GHz CPUs! Smoking! I remember we had issues with this when I set it up, but since it would be headless in excess of 99% of the time, I decided to ignore the issues. Which are now coming to bite me in the rear.

This is a pain because I have the web server, with lots of content here and I don't want to chance a hard power down. No problem Bill, we'll call my friend by a pseudonym, says we can use his old laptop which is lying around on the Sun box we have there. So we open it up, plug it in and boot up. It's running Fedora, so I'm asking him is he sure we can get in, but he assures me he has a RDP client to connect to terminal services.

After 2 minutes, it's still booting, but there are issues. Some type of file system problem and it's checking blocks. Great.

A few more boots and 15 minutes later we decide this laptop is dead for the time being. Now what? I'm considering doing a hard shut down. I thought about calling Andy and asking him to remotely shut it down, but didn't really want to get laughed at for 15 minutes. Still I'm about to when Bill remembers the other Dell that's on the shelf is running W2K. It's old as well, a PII of some sort, but we log in (painfully slowly) and terminal server over and shut down the box.

After packing up the servers, getting cables, the firewall, double checking we have the power adapters for the switch and firewall, and a few other things, we get loaded up and head back down to Denver.

Saturday, 2:00pm

We drive back down and as I'm taking the exit off I-25 I get a call. Let's stop by Anthony's. Grumble grumble, but I do it. For those of you not in Denver, Anthony's is a New York style pizza place that's consistently voted one of the best in Denver. And I happen to agree it's one of my top 2. So we stop for a quick bite to eat, which isn't that quick. Anthony's not there and the twenty somethings running the place aren't that quick. And Oklahoma is playing Oregon, so we watch a few sets of downs.

We get to the data center and the first thing is to get Bill badged. Since he lives up North and is my backup, I wanted to get him access. Since he's rarely down here and I need to approve the badge, we decide to do that first. That's a nice 15 minutes and then we get our booties on and check out things. We also wanted to talk about where to put things.

I've been in here before, having setup the mail server, but I wanted to grab a crash cart with monitor and keyboard as well as the tool cart so we were ready. I also like to be sure there's no one working right there next to us or that might cause us issues as we bring in equipment. There isn't, so we load up a cart with servers and bring it in.

The cabinet only has a single shelf in it, on which the mail server sits. I requested another shelf when we badged Bill and it appears just as we're straightening the first shelf. The guy that help me put it in screwed just one bolt in each brace and they weren't lined up. I was holding the shelf and didn't notice until he'd left and decided not to deal with it that day. Still since we were both here, this was a good time to do it right.

We get both shelves in and then stack up the servers. I'd forgotten the rails, actually I'd forgotten to look for the rails. Not sure if I even have them anymore (I'd acquired 2 sets of rails when my startup failed), but in any case, stacking 2 servers isn't a big deal. As I plug in servers and KVM cables, Bill mounts the firewall and switch and gets them powered up.

It's almost 3 and I'm thinking we're about done. Plug in some cables, change the firewall IP and we're golden. I should be smacked on the head for expressing such thoughts, which I'm sure I did express out loud. How else could I explain things moving south in a hurry from here.

At this point we plug in the mail server to the switch and the Internet connection to the switch. And it doesn't work. No link from the ISP side.

"No problem, " I say, "flip the MDX switch."

Bill patiently proceeds to lecture me on the high quality and the vast capabilities of his high end switch. Apparently these switches from Cisco don't condescend to have something as lowly as an MD/MDX switch. I mumble something about this being a piece of sh_

"Hey!"

I let it go as we search for a crossover cable to connect it. After a few minutes of comparing ends of each cable we've brought to check for a crossover, I take another deep breath and mumble again about my $49 switch at home having the ability to connect to other switches out of the box as we lock everything up and leave. Locking up is fun since we get to disconnect the crash cart from the KVM, put the doors back on the cabinet and lock them down, a nice 5 minute process.

Oh well, I get to run home and see the kids. I'm only 3 miles from the data center down a back road, so we decide I'll go grab a couple more KVM cables so they're ready for any more equipment as well as see the kids. Bill's going to Microcenter for a couple longer patch cables and a cross over. He's the kind of guy that likes things organized, wants cables color coded, doesn't like keyboards resting on the floor, etc. So he'll just buy a few to fit the cabinet. We walk out, discard the booties and I head for a quick restroom break.

I come out and Bill's standing there. Strange since I thought he was going to Microcenter. So let me tell you about Colorado. It's a high desert near Denver, dry, sunny more of the time, and hilly. As in there is relatively little flat land, which includes the parking lot at ViaWest. Bill had parked his van in an incline and being low on gas, it wouldn't start.

So he got to come with me back to the house and grab my gas can, which had a couple gallons in it. My 3 year old ran up and jumped in my arms as we arrived, so it took about 20 minutes to get a few cables, the gas, and extricate myself from my daughter, who thinks she's a horse and wanted me to feed her bits of apple from my hand "like a horse". Dodging a dark look from my wife (it was 3:30 and you remember my estimate), we jumped back in the truck and drove to ViaWest. A few gallons later, Bill's van started and we headed to Microcenter since there wasn't much for me to do at ViaWest, Bill's a geek and could get lost at Microcenter, and, well, I'm a geek to, so I can always browse for a few minutes.

So we're driving down the back roads near Cherry Creek State Park to loop around the traffic in the tech center. It's a 35mph road, slightly out of the way, but it avoids the 15 traffic lights we'd hit going the main way. I've found it quicker as one of the few drawbacks to living in Denver is the traffic. So we're cruising, shooting the breeze when a bumblebee blows in Bill's window and sticks at the base of the windshield. It's a big bee, like 2 inches long and thick, but he's just sitting there. As I get ready to roll down my window and look for something to shoo him out I see Bill. He's leaned back into his seat, so far that his rear is no longer on the seat, apparently attempting to get as far as possible from the bee. Just as I'm about to ask him what he's doing, the van swerves and decelerates quickly onto the shoulder and Bill hastily unbuckles and jumps out the door. I'm left sitting there staring at an empty seat. I look back at Bo, Bill's akita that's riding the back, and I swear he shrugs at me, so I get out.

Bills breathing a little quick standing about 5 feet behind the read of the van. I ask him what's wrong, though I have a good idea what it is.

"I'm terrified of bees." I get breathlessly.

Now it may seem funny, but you really have to get the picture here. Dust is still swirling in the air from the sudden stop, Bill is breathing hard, standing more than an arm's reach to the read of his mini van, which is rather long. The bee is more than 15 feet away, at the base of the windshield. He's much more likely to meet one of the bees friends flying along the road than that particular bee. Also Bill is a former football player at the college level, well over 6 foot and 200 lbs, he's something of an intimidating sight. So it's a good thing I'm a friend as I just started laughing. I kidded him quite a bit before I admitted that I'm the same way around snakes. After shooing it out the window, with Bill 4 or 5 feet behind me, we get back on our way. Another 15 minutes blown.

As you might expect with 2 geeks, the Microcenter trip isn't the quickest one, but we do get out of there with 4 cables in less than 45 minutes. And less than $50, probably a record for both of us.

Saturday, 4:30pm

We get back to ViaWest, plug in the switch with the crossover cable and the link lights up.

And goes dark. This is starting to be really fun at this point. We call the NOC and I let Bill talk to them. It goes something like this, my unspoken thoughts in parens:

"Hi, we're with SQLServerCentral.com (like that carries weight) and we're having trouble connecting to the switch"

"It's a Cisco blah blah blah (like I remember the model). I've had similar problems in the past."

"Yes, we're set to 10 full, can we switch to auto? I've seen the Cisco products work much better with Auto when talking to each other."

We try that and it doesn't work. Now I trust and respect Bill, he's one of the best network engineers I've worked with. He passed 1/2 of the CCIE before his job changed and he lost interest. Very few people I've met know more about networking that he does. Since that's the case, my role in this part of the adventure is cable boy. I unplug ethernet cables from the switch and plug them back in when I see Bill nod. Occasionally I get to more the serial like from the switch to the firewall and back as well.

"No still no link. I'm not seeing the gateway. Are you sure you're on auto? I've got this side set to auto auto."

A few minutes pass.

"What's that? There's no spanning tree enabled here.

"This is a simple setup. The switch isn't seeing your switch. No firewall, nothing else in the way."

"I'm telling you, there is no spanning tree enabled on any port. I'll check the config again (a sh run on the switch does this). There, port 22, no spanning tree."

"Trunking? There's no trunking here."

"I'm telling you, VTP is not setup. This is just a simple switch set with 2 VLANs"

And on and on. For about 90 minutes I'm listening to this, plugging and unplugging based on nods from Bill and the context of the conversation, or at least the half I'm privy to. Finally we've had enough. Bill gives up on the switch and as he tries to explain the simplicity of the switch to me, unplugs the switch and sets up the firewall to plug directly into the cabinet. We'll remove the VLANs and run everything behind the firewall. Not a big deal.

So he says as we remove the VLANs and cycle the power on the switch. Now the firewall is up and we set it's IP and it connects right in. Now we spend a bunch of time in a terminal session removing rules and routes that are not needed since the front side of the network has changed. We do at least save off the old rules since we need the same ones with different IPs, so it's good to know which ports and which redirections we have setup. The idea being to minimize the changes behind the firewall.

So as Bill removes things and I watch over his shoulder, we get the rules back in and move to test the first box. No connection, so we try the second. No connection. Finally we try to newly moved db server and we connect. This is a new IP, and Bill thinks of something right away. As a test, we remove the routes and lines from box 1 and add them back with a new external IP, one above box 3. It works, so we're back on the phone.

"Hi, it's Bill again, I was talking to you a bit ago about the switch. Yeah, right, so I'm having some issues with the first couple IPs."

"I need you to clear the ARP table."

"No, it's a problem with the arps not clearing. The MAC that this IP was bound to was changed since we moved it behind the firewall, so I need you to clear out the ARP table."

"No just clear the table. I don't even know why you have the timeouts set so high. It's not like you have thousands of boxes here. I just need the ARP table cleared so the switch will rebind to the new box."

This goes on for about 10 minutes and finally the guy gets the ARP table cleared and we can connect to box 2 again. I test the site and it's back up, not bad considering it's after 7:00. We quickly remove the rules for Box 1 and put them back on the IP we'd already setup. Now we check connectivity for all boxes. For some reason when I built the mail server, I named it the same as the database box. They were built like six months apart and were never on the same network, so I never noticed, but now I have a Windows conflict. So I rename the mail server, and reboot both boxes. There's a little config to do, but I want to get the rules straightened out first, so I relinquish the servers to Bill. After some double checking, he gets all the rules in and goes to connect his Sun box, which runs a few small services for another client.

I do some configuration on the mail server, rename the SQL Server, change a couple of IPs in the software and it still won't start the mail software. I recheck the site from a remote terminal server box and it's up, so I'm mostly OK. I post an answer to a question, run through a few other items and it appears we're back online. Bill then tells me he needs to take the database server offline for a minute because I used the wrong color of ethernet cable to connect it to the switch. Red is for DMZ boxes only, blue for others. Grumble, grumble, go ahead, I say.

It's getting cold and I'm freezing at this point. Everything seems to be working except the mail server, but it's been a few hours in a cold room and I'm ready to go. We double check the connections, ensure the serial connection is still on the firewall for remote config, look at the site once more and then lock up.

A short drive gets me home where I grab a sweatshirt first thing as I'm freezing. As I warm up, we log in from my machine and everything's there. I realize the DNS for the mail server got moved and I change that since there's a chance that's the issue. We give that some time and I spend a few minutes with the kids.

That's it for the move, or at least those are the marching orders from my wife, so I say good night to Bill, extract him from my middle son's arms, and then put the kids to bed. In another moment of clarity at 6am while I'm making my daughter's bottle, I realize that the mail server needs to recognize the new name, so I find the config after dropping the bottle off with my little girl, see it's trying to connect with the old name and change it and Shazam!, everything is back online.

Conclusion

What did I learn? Well first, not to tell my wife my estimate, but instead double it and then add 2 hours.

I also learned that even a simple move can have lots of complexity. I'd mistakenly assumed that since I had one box up there, things would be quick. I should have expected that moving the firewall would account for some issues. If some time had been taken to think through the move, I probably also would have staged a few more pieces of equipment, like cables and such. Having configurations and IP assignments and such written out ahead of time as well would have made things smoother. Lastly, Bill and I should have decided early on to use 2 switches or move everything behind the firewall. I don't doubt that VLANs are the better way to do this, but to me being simple and clean is better than a more elegant solution. I should put a spare switch in there as well since it's not much money and having it handy would be preferable to another trip through bee-infested Cherry Creek park.

Still overall things went well and nothing "broke" during the move. All parts are intact and things are looking good on the site.

©dkranch.net 2004

Return to Steve Jones' home