June 22, 2011 at 9:08 am
We have a very strange RS 2008 issue. The strange part is how we get around the problem as I will describe below:
First, the problem:
1. RS 2008 (64-bit) runs for awhile hardly using any CPU (1%), serving reports just fine
2. A certain order will be hit by a user that will never return (all reports are called via ASP.Net ReportViewerControl)
3. This spinning report will eat up 50% of CPU (2-CPU machine). The CPU will stay at that level
4. During this time, other orders using the same report process just fine (makes you think this is a data related problem with the specific order, but, go read the work-around below.
5. If this same order report is attempted again or if another order causes the other thread spin, the RS CPU goes to 100% and stays there. The only way out now is to restart RS.
Here's the strange workaround:
1. Make an insignificant change to the group on the Tablix (the report has one Tablix that has a recurring group for each line item on the order). Basically I either add or delete a blank row from the Tablix group.
2. Re-Deploy the report. Now that order will work just fine from now until infinity. But the spinning report will happen again on another order. Then, I will just do the same workaround, add a blank line to the Tablix, redeploy and it will work.
One other clue: When I run the report using the spinning order under BI Studio, it does return but it returns with a "Out of Memory" exception. Of course, if I change the Tablix by adding a blank row, the problem is solved.
I speculate that the spinning report is eating all the memory causing the GC to run and eat the CPU (just a theory at this point). I also speculate that the Tablix must be getting corrupted internal to RS in some RS cache that is holding the cache based on the values of the parameters (because the report will work fine for other parameters while the "bad" report is spinning.)
We have a case open with MS on this, but after 1/5 weeks, they can't determine the issue from the several dumps, etc that I have uploaded.
Anyone have a clue to what this is because MS doesn't?
June 22, 2011 at 9:47 am
was this report written in 2008 or upgraded from 2005?
June 22, 2011 at 10:13 am
Written in 2008 just recently
June 22, 2011 at 10:15 am
if you run the script in Management Studio do you ever run into the same problem?
June 22, 2011 at 10:18 am
inside of BI studio, I get an out of memory exception but can fix the problem in th esame way - add blank line ot tablix and rerun in Preview mode, then it works in BI also.
June 22, 2011 at 10:21 am
But if you take your script out of BI Studio and run it in Management Studio? Does it every run out memory in management studio?
June 22, 2011 at 10:23 am
Do you mean the underlying stored procs? These do run fine in Studio with no problem.
June 22, 2011 at 10:40 am
Ok, I was wondering if it was something like parameter sniffing. When the reports go AWOL have you tried looking at the SP call in profiler or using the DMVs to look at the execution plan?
July 22, 2011 at 5:42 am
OK, finally determined that this is a bug in RS 2008. Has been fixed in 2011 (Denali) according to Microsoft.
The workaround is to set the InteractiveHeight = 0in. and all reports.
From the dump reads, the bug is causing the renderer to mis-calculate the # pages up into the 60,000-70,000 range even for reports that are just 1 page long. Thus, the renderer runs until it gets a ReportServerStorageException, all the while eating up the CPU and the CPU never gets released.
A very severe bug for PRODUCTION.
July 22, 2011 at 5:47 am
bddupree (7/22/2011)
OK, finally determined that this is a bug in RS 2008. Has been fixed in 2011 (Denali) according to Microsoft.The workaround is to set the InteractiveHeight = 0in. and all reports.
From the dump reads, the bug is causing the renderer to mis-calculate the # pages up into the 60,000-70,000 range even for reports that are just 1 page long. Thus, the renderer runs until it gets a ReportServerStorageException, all the while eating up the CPU and the CPU never gets released.
A very severe bug for PRODUCTION.
WOW, do you have the link to the full KB?
July 23, 2011 at 3:40 pm
There isn't a KB, not yet anyway. We had a case open with MS. Once we found the workaround, we didn't press MS for a fix because the workaround is sufficient for us.
MS did say they have fixed this in 2011.
July 23, 2011 at 5:28 pm
bddupree (7/22/2011)
OK, finally determined that this is a bug in RS 2008. Has been fixed in 2011 (Denali) according to Microsoft.The workaround is to set the InteractiveHeight = 0in. and all reports.
From the dump reads, the bug is causing the renderer to mis-calculate the # pages up into the 60,000-70,000 range even for reports that are just 1 page long. Thus, the renderer runs until it gets a ReportServerStorageException, all the while eating up the CPU and the CPU never gets released.
A very severe bug for PRODUCTION.
We're a wee bit behind on the power curve and just started moving things over to 2K8 R2. We haven't run into this, yet, but it sure will be handy to know when it does happen. This one's going into my "briefcase". Thanks for taking the time to post back.
--Jeff Moden
Change is inevitable... Change for the better is not.
Viewing 12 posts - 1 through 11 (of 11 total)
You must be logged in to reply to this topic. Login to reply