October 14, 2008 at 11:34 pm
Hi guys,
I'm seeing CXPACKET wait types in one of my database running SQL2005 which causing the CPU hit 100%. The server is 64 bit Windows 2003 with 8 CPU. Several tables contain over 10 million rows. Does anyone encounter this issue before and what was done to fix the problem?
Thanx.
October 15, 2008 at 1:24 am
CXPacket is wait caused by a parallel skew. What happens is that SQL runs a query in parallel and, for whatever reason, one of more of the threads lags behind. The ones that finish have to wait for the slower ones to catch up. That wait is the CXpacket wait
You can either reduce the max degree of parallelism for the entire server or, if it's a specific query that's giving the problem, you can add the maxdop hint to that query.
Be careful if you do the former, as it will affect the entire server. If you decide to go that route, I would suggest setting the server's max degree of parallelism to 4 (half the number of cores that you have) and see how it goes from there.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
October 16, 2008 at 2:15 pm
Hi everyone I am having a similar problem here, well on my dev server I have 4 cpu, I dunno why I get so much CXPACKET wait time, well my maxdop is 0, should I set it to 2? so I can get a better performance?
Thanks for your help,
October 17, 2008 at 8:03 am
At least 75% of my clients have multi-cpu servers with insufficient RAM or IO capabilities (or both!), and thus suffer from significant CXPACKET waits on largish queries. It is often optimal in general in situations like that to simply set MAXDOP at the server level to some fraction of the total number of CPUs (1/4 to 1/2 usually). In other cases you can specify the maxdop as an OPTION for an individual query.
NOTE: if you have hyperthreading enabled it gets MUCH worse. Consider disabling HT if you are seeing logs of CXPACKET waits first, then reevaluating and if necessary adjust MAXDOP.
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
October 17, 2008 at 8:46 am
chileu17 (10/16/2008)
Hi everyone I am having a similar problem here, well on my dev server I have 4 cpu, I dunno why I get so much CXPACKET wait time, well my maxdop is 0, should I set it to 2? so I can get a better performance?
I would at least try it to see if it helps.
[font="Times New Roman"]-- RBarryYoung[/font], [font="Times New Roman"] (302)375-0451[/font] blog: MovingSQL.com, Twitter: @RBarryYoung[font="Arial Black"]
Proactive Performance Solutions, Inc. [/font][font="Verdana"] "Performance is our middle name."[/font]
October 20, 2008 at 8:42 am
TheSQLGuru (10/17/2008)
At least 75% of my clients have multi-cpu servers with insufficient RAM or IO capabilities (or both!), and thus suffer from significant CXPACKET waits on largish queries. It is often optimal in general in situations like that to simply set MAXDOP at the server level to some fraction of the total number of CPUs (1/4 to 1/2 usually). In other cases you can specify the maxdop as an OPTION for an individual query.NOTE: if you have hyperthreading enabled it gets MUCH worse. Consider disabling HT if you are seeing logs of CXPACKET waits first, then reevaluating and if necessary adjust MAXDOP.
Thanks we are gonna try that out because this CXPACKET wait is just killing our performance
October 20, 2008 at 11:16 am
Probably the BEST thing you can do (if able) is to a) max out the servers RAM (which is hopefully many, many GBs) and b) add more spindles to the IO mix to improve IO performance.
I also recommend (regularly on performance forums such as this) that you get a professional to come in (onsite or remote) and give your systems a performance review. There are a pleathora of things that can be done suboptimally that will hurt performance. 🙂
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
October 20, 2008 at 1:49 pm
yeah we are thinking on calling some MS performance expert on this matter, because we don't seem to get a better performance in any of our attempts. But just to try something new, what did you mean with more spindles to the IO mix to improve IO performance?
Thanks for the help guys 🙂
October 20, 2008 at 4:03 pm
chileu17 (10/20/2008)
yeah we are thinking on calling some MS performance expert on this matter, because we don't seem to get a better performance in any of our attempts. But just to try something new, what did you mean with more spindles to the IO mix to improve IO performance?Thanks for the help guys 🙂
CXPACKET waits are due to parallelism where the query optimizer decides to break up the IO and processing for a query into multiple threads that run concurrently then it gathers the data back together to provide the necessary output. In pretty much every case, the CPUs wait for IO streams because the IO subsystem is slow as molasses in February, relatively speaking. 😀 So by providing better IO capabilities you can feed those GHz-speed CPUs with data with fewer waits --> much faster parallel query performance. This is the 10000 foot view, btw. There is just a weeeee bit more to it than this. 😎
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
October 21, 2008 at 3:23 pm
Sorry SqlGuru :hehe: I think I couldn't explain to you my question. I do understand your point, BUT how do I implement what you are talking about?More indexes?new transactions?I know that the most probable answer is that it depends on every case right lol well I have a dedicated server with no other apps running on it. It runs on Windows server 2003 with a HD of 150 gb with a redundant SAN. We have installed a RAID 5 on it 🙁 please any suggestions are welcome. Thanks in advance again.
October 21, 2008 at 5:54 pm
chileu17 (10/21/2008)
Sorry SqlGuru :hehe: I think I couldn't explain to you my question. I do understand your point, BUT how do I implement what you are talking about?More indexes?new transactions?I know that the most probable answer is that it depends on every case right lol well I have a dedicated server with no other apps running on it. It runs on Windows server 2003 with a HD of 150 gb with a redundant SAN. We have installed a RAID 5 on it 🙁 please any suggestions are welcome. Thanks in advance again.
Hmm, I thought that "add more RAM" and "add more physical spindles" was pretty self-explanatory. 😀
It really is that simple though. Get data through the CPUs more quickly (which RAM and IO perf boost will do) and CXPACKETwaits drop down.
Other things: cut read requirements by a) better indexing b) refactoring query to reduce data pulled/accessed, lower maxdop (already covered in depth), ensure all IO config options are set right between sql server and SAN (HBA [queue depth?, cache ratios and size?), SAN cache, drivers up to date?, etc). There are other things as well.
Best is to get a pro to come in and give your system a performance review. We have hardly touched the surface here.
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
October 22, 2008 at 6:03 pm
Well-tuned queries - in an OLTP system - genarally should not parallellize. This is almost certainly a case of a suboptimally tuned system.
Is auto-updatestats, auto-createstats turned on for your databases?
Do you regularly carry out index defragmentation?
Are your queries properly indexed (as was already mentioned)?
Is your tempdb properly configured?
These are some of the questions to consider.
__________________________________________________________________________________
SQL Server 2016 Columnstore Index Enhancements - System Views for Disk-Based Tables[/url]
Persisting SQL Server Index-Usage Statistics with MERGE[/url]
Turbocharge Your Database Maintenance With Service Broker: Part 2[/url]
October 22, 2008 at 6:11 pm
Also, gather some of your largest tables, those most actively queried in your system.
Do "sp_helpindex tblName" to get a list of indexes for each table.
Run "DBCC SHOW_STATISTICS ('tblName', 'idxName')" on the tables and select indexes.
Is the Rows Sampled equal to the Total Rows in the index?
If not, you may want to run "UPDATE STATISTICS tblName WITH FULLSCAN" to ensure stats is maintained on the table on the basis of the full data distribution (as opposed to a subset sample). I have often found this to solve my performance problems.
__________________________________________________________________________________
SQL Server 2016 Columnstore Index Enhancements - System Views for Disk-Based Tables[/url]
Persisting SQL Server Index-Usage Statistics with MERGE[/url]
Turbocharge Your Database Maintenance With Service Broker: Part 2[/url]
June 30, 2009 at 6:35 am
WE had an IO problem on RAID 5. Change to RAID 1 0 if possible. It will improve IO alot. Add more physical drives to the SAN drive. Put the log files on a different SAN drive with different physical drives.
Also get your SAN provider to diagnose the SAN. We have a cache problem that fills up on the controller and then slows the SAN performance drastically.
This will all improve your IO capabilities.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Do not reinvent the wheel.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
July 20, 2009 at 7:21 am
I've seen this problem more often on SQL 2005 than on SQL 2000 instances, but I must agree servers tend to get more cores everyday. On some instances used mainly for OLTP I set the maximum number of processors per query to one, thereby effectively disableing parallel queries. This sometimes improves both query performance and throughput but it is not an options if you want to perform some serious reporting on a server.
But there is no general solution. I/O seems to be the bottleneck but in some cases I could greatly reduce query execution time (and CXPACKET waits) by using the OPTION LOOP JOIN. No two cases are the same and if a SAN is involved, things are getting more complicated because most SAN experts are not used to the demands of SQL server regarding caching, write order, and many other aspects of disk I/O. Getting a SQL server up to speed on a SAN is far from simple, and requires a specialist with knowledge and experience on both subjects.
As mentioned by others, up-to-date statistics won't ever harm your queries. More memory does not always improve performance; a very large table might still not fit into memory, and thus a table scan on this table might still require massive disk I/O. In this case profiling will offer you some insight in missing indexes that could increase performance, but as always, it depends ...
Viewing 15 posts - 1 through 15 (of 37 total)
You must be logged in to reply to this topic. Login to reply