June 25, 2015 at 3:23 am
Hi, I'm having some challenges around latency on our Dell Compellent SAN and I thought I would post this to see what others take on this is. I'm currently seeing read latency anywhere around 30 ms to 70 ms, we don't really pressure our system much, MS MAPT is telling me we are using around 1500 IOPs for each of our 4 SQL server instances across 3 VM's, this workload is spread out somewhat with overnight batching on 2 of these instances. We are an iscsi (10GB) environment and have a "application pool" dedicated to SQL server capable of 17000 IOPs for 4 hours with the tiers we have set up, 5 SSD fast write, 5 SSD fast read, and a couple of trays of 10K disks to build up the rest of IOPs over time as blocks need to move down. We seem to only be tickling our SAN in terms of performance but the read latency seems to be abysmal, as I say it jumps up and down constantly, writes are pretty consistent at around 2-4ms. All running Server 2012 R2 and SQL Enterprise 2012 SP2 on HP C7000 G7 and G8 blades with 10GB back to SAN switch, I know this is a pinch point as we only have one card, with 4 10GB connections in active passive, 2 to LAN, 2 to SAN). We also use Dell force10 S4810 switches dedicated to SAN switching.
I have a separate server running my test rig with separate 10GB iscsi link to SAN switch to eliminate any networking issues here running server 2012 R2 Hyperv, same SQL servers clustered as we do in production with a VM running dynamics CRM. When I hit this CRM server and call up a record it jumps to 60/70 ms this is all looking at Avg. Disk sec/Read counter and write counter, but as i say writes seem to be fine. I have tried a couple of different NICs in this server, both Broadcom and Intel but they are both around the same...
I have read somewhere that when dealing with low IOPs latency can be a bit overly sensitive..?
Thoughts?
June 25, 2015 at 7:11 am
What is the analysis of your storage Admin? Usually read should not be high? Does it happen on all server using same SAN? If it is new server with no data you can try using SQLIO utility to investigate further.
HTH
---------------------------------------------------
"Thare are only 10 types of people in the world:
Those who understand binary, and those who don't."
June 25, 2015 at 12:15 pm
One thing you might want to do at night with no load is to drive the SAN I/O through the roof with some kind of giant test query that gets run on multiple machines and just look at I/O Waits and Queue Waits via Perfmon. You might want to set up some different scenarios and use the results as a baseline at differing levels of I/O activity. This might help illustrate where the "choke point" really is.
Steve (aka sgmunson) 🙂 🙂 🙂
Rent Servers for Income (picks and shovels strategy)
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply