Once it happened quite a black Friday for my environment after a deployment. The application got sucked so that becoming worst and worst. The average CPU raised over 80% (Figure 1). I just captured the moment and want to share.
Figure 1. Very high CPU on a 24 core with 128GB RAM
One of the bad queries found during the performance tuning is the next one:
SELECT TOP (@ProcessBatchCount_p) * FROM dbo.Queues AS Q WITH (NOLOCK) WHERE ((Q.QueueTypeID NOT IN (@queueTypeBetSlipWon_p, @queueTypeBetSlipCancelled_p, @queueTypeBetSlipRejected_p, @queueTypeBetSlipRefoundCancelled_p, @queueTypeBetSlipEndGame_p) AND Q.AttemptCounter < @attemptCounter_p) OR (Q.QueueTypeID = @queueTypeBetSlipWon_p AND Q.AttemptCounter < @winAttemptCounter_p) OR ((Q.QueueTypeID IN (@queueTypeBetSlipCancelled_p, @queueTypeBetSlipRejected_p, @queueTypeBetSlipRefoundCancelled_p) AND Q.AttemptCounter < @refundAttemptCounter_p)) OR (Q.QueueTypeID = @queueTypeBetSlipEndGame_p AND Q.AttemptCounter < @endGameAttemptCounter_p)) AND Q.QueueTypeID NOT IN (SELECT Item FROM #resultsQueueType) AND Q.QueueTypeID NOT IN (SELECT Item FROM #betSlipItemsQueueType) AND (Q.ProcessOn IS NULL OR Q.ProcessOn < @UtcTimeNow) ORDER BY Q.ProcessOn;
The problem was identified as overloaded application queues that were processing XML files. The instance was running this code in 20 databases simultaneously and it practically happened a complete blocking.
Here are some advice:
- DO NOT USE functions in comparison unless you must to. Try to avoid.
- DO NOT USE many @parameters in queries.