Slow site

+1 and has been for the last few hours

+1000... there has to be a word worse than excruciating over the last couple of days. It's not consistently bad but it's been absolutely killer slow for extended periods over the last couple of days.

We are looking. We had some strange networking problems between the DB and website. We have a ticket open with the colocation facility

Hi folks,

Really sorry about these issues - we're well aware of them, and working with the hosting company on them.

Basically, we're seeing intermittent packet loss between the SQL boxes and the front-end nodes (up to 10% with 1kB packets!), which is having the predictably devastating effect on performance.

As of 1415 UTC, they have switched the path between the nodes to an alternate route, which is so far looking much healthier (no loss, and lower latency).

We'll carry on watching for it, and carry on chasing appropriately...

Cheers,

Rob (Redgate IS team)

Thanks for feedback...at least we now know you are on the case....hopefully you will be able to resolve the issues quickly.

At time of writing....site response seems good for me

From a monitoring point of view, it looks like things have remained stable since that change. If anyone is still seeing otherwise, please pop something on this thread.

Thanks,

Rob

This was starting to sound like a cable gone bad or someone setting a router to auto-negotiate or half duplex.

Thanks for looking into it. It was pretty brutal this morning.

Bother - the issue is back as of a few minutes ago. Average 15% packet loss again starting 2052 UTC.

Sorry all, normal service as soon as we can :-(.

Cheers,

Rob

For the curious, here's what the network traffic currently looks like between the two boxes:

(Dark blue line is packet loss, light blue round trip time)

The latest we have from the hosting provider is that one of their links is "saturated with customer traffic" (not ours!), and that we are not the only ones affected. Hopefully they'll get additional capacity sorted as soon as possible; if it's looking like it's going to go on for an extended period we will look at whether we can work around the issue any other way.

Cheers,

Rob

Very slow at 18:46 EDST. Ya just gotta love relying on 3rd parties. This and other problems we've seen over the last 5 years is why we don't do "Cloud" or 3rd party hardware providers. I do some part-time work for some companies that do and it's been a comparative nightmare not worth any potential cost savings.

True, though comparatively speaking, on average I'm sure they do a better job than if we were to host SSC in Redgate Towers - not least, our internet connections and mains supply goes down a lot more often than at a proper datacenter ;-).

Ultimately, unless you're big enough to be peering at multiple point of presence with physically diverse paths and so on, you're probably placing yourself in the hands of some collection of third parties. By and large, these folks have done us well over the years - it's been a very atypically bad week for them, so as frustrating as it is, I'm optimistic they will get it sorted and back to normal soon.

Ironically, while watching this, my home DSL line has been showing similar packet loss for completely unrelated reasons, Dropbox has had issues, and so have ZenDesk. Obviously the internet got out of bed the wrong side today!

Cheers,

Rob

Heh... good grief. Between 20 and 60 seconds between every click now (10:30PM EDST). This is brutal. I sure hope you good folks can get to the bottom of this soon. Signing off for the night.

I was seeing the same thing at the same time. I was up to > 90 seconds to reply and gave up. Now (10:56 EDT) it's still around 60 seconds to bring up the page to submit a post.