October 23, 2006 at 3:36 pm
Hi
We're having trouble with memory configuration in Reporting Services 2k5 SP1, running on Win 2k3 SP1. We have a very large report, which we've been running to benchmark performance.
First, here are the results of our testing:
The pattern we noticed with the first three steps, was that Reporting Services isn't using more than 80% of the physical RAM available on the server:
Following the directions in "Configuring Available Memory for Reporting Services" (http://msdn2.microsoft.com/en-us/library/ms159206.aspx), we tried increasing the MemoryLimit of both the Web Service and the Windows Service.
We changed the memoryLimit in machine.config to:
<processModel autoConfig="true" memoryLimit="120" />
and the memoryLimit in rsreportserver.config to:
<MemoryLimit>120</MemoryLimit>
<MaximumMemoryLimit>140</MaximumMemoryLimit>
It looks like the ReportServer is reading this, as when it is restarting, the ReportServer_*.log file records:
w3wp!library!1!11/10/2006-10:20:56:: i INFO: Initializing MemoryLimit to '120' percent as specified in Configuration file.
w3wp!library!1!11/10/2006-10:20:56:: i INFO: Initializing MaximumMemoryLimit to '140' percent as specified in Configuration file.
However, it doesn't go above this 80% threshold.
More concerning is the System.OutOfMemoryException exception described above. Following the suggestions in KB 909678, we've tried tweaking the MemoryLimit and there is clearly still plenty of RAM left available on the server. This is happening repeatedly when we try to run four instances of the report concurrently.
Are there other parameters we should be adjusting to ensure Reporting Services uses the available RAM?
Thanks,
Andrew
October 24, 2006 at 5:28 pm
curious, you are restarting RS... are you also restarting IIS?
Thank-you,
David Russell
Any Cloud, Any Database, Oracle since 1982
October 24, 2006 at 6:30 pm
Am also curious about this issue. When you say the report is 'large' just how many rows are you trying to retrieve? What amount of formatting/logic is contained in the body of the report? Have you looked at your execution and trace logs to determine whether the bottleneck is occuring during execution or rendering? You mentioned you'd looked at the KB article -> as it suggests did/have you tried running the report but retunring it as csv (assuming you were trying for html previously)?
Steve.
October 24, 2006 at 6:57 pm
Hi David
We're rebooting the whole server to add the new RAM, so yes, I am restarting IIS.
Thanks,
Andrew
October 24, 2006 at 10:08 pm
Hi Steve
When I say the report is large, I'm not kidding - the stored procedure returns about 180,000 rows.
The report contains a single matrix control, which does a GROUP BY, so I'm expecting the rendering to be fairly intensive on machine resources, however I rather it didn't just fall over!
Based on this excerpt from the ReportServer_<timestamp>.log, it looks like it dying during the rendering:
w3wp!library!1!10/25/2006-16:43:57:: i INFO: Call to RenderFirst( '/Testing/Monthly Pay Detail' )
w3wp!library!5!10/25/2006-16:43:59:: i INFO: Call to RenderFirst( '/Testing/Monthly Pay Detail' )
w3wp!library!7!10/25/2006-16:44:23:: i INFO: Call to RenderFirst( '/Testing/Monthly Pay Detail' )
w3wp!runningjobs!8!25/10/2006-16:44:30:: i INFO: Adding: 2 running jobs to the database
w3wp!library!8!10/25/2006-16:45:00:: i INFO: Call to RenderFirst( '/Testing/Monthly Pay Detail' )
w3wp!runningjobs!9!25/10/2006-16:45:31:: i INFO: Adding: 2 running jobs to the database
w3wp!processing!1!10/25/2006-16:52:20:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: An unexpected error occurred in Report Processing., ;
Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: An unexpected error occurred in Report Processing. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.ReportingServices.ReportProcessing.MatrixInstance.AddCell(ProcessingContext pc, NonComputedUniqueNames& cellNonComputedUniqueNames)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateOutermostSubtotalCell(MatrixInstance matrixInstance)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateSubtotalOrStaticCells(MatrixInstance matrixInstance, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, Boolean outerGroupingSubtotal)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixHeadingsObj.CreateInstances(RuntimeDataRegionObj outerGroup, ProcessingContext processingContext, MatrixInstance matrixInstance, Boolean outerGroupings, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, MatrixHeadingInstanceList headingInstances, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateInnerGroupingsOrCells(MatrixInstance matrixInstance, RuntimePivotGroupRootObj currOuterHeadingGroupRoot)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixHeadingsObj.CreateInstances(RuntimeDataRegionObj outerGroup, ProcessingContext processingContext, MatrixInstance matrixInstance, Boolean outerGroupings, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, MatrixHeadingInstanceList headingInstances, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateInstance()
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.Traverse(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.TraverseAllLeafNodes(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupingObj.Traverse(ProcessingStages operation, Boolean ascending)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupRootObj.CreateInstances(ReportItemInstance riInstance, IList instanceList, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixHeadingsObj.CreateInstances(RuntimeDataRegionObj outerGroup, ProcessingContext processingContext, MatrixInstance matrixInstance, Boolean outerGroupings, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, MatrixHeadingInstanceList headingInstances, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateInstance()
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.Traverse(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.TraverseAllLeafNodes(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupingObj.Traverse(ProcessingStages operation, Boolean ascending)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupRootObj.CreateInstances(ReportItemInstance riInstance, IList instanceList, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixHeadingsObj.CreateInstances(RuntimeDataRegionObj outerGroup, ProcessingContext processingContext, MatrixInstance matrixInstance, Boolean outerGroupings, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, MatrixHeadingInstanceList headingInstances, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateInstance()
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.Traverse(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.TraverseAllLeafNodes(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupingObj.Traverse(ProcessingStages operation, Boolean ascending)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupRootObj.CreateInstances(ReportItemInstance riInstance, IList instanceList, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixHeadingsObj.CreateInstances(RuntimeDataRegionObj outerGroup, ProcessingContext processingContext, MatrixInstance matrixInstance, Boolean outerGroupings, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, MatrixHeadingInstanceList headingInstances, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixGroupLeafObj.CreateInstance()
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.Traverse(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupLeafObj.TraverseAllLeafNodes(ProcessingStages operation)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupingObj.Traverse(ProcessingStages operation, Boolean ascending)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeGroupRootObj.CreateInstances(ReportItemInstance riInstance, IList instanceList, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixHeadingsObj.CreateInstances(RuntimeDataRegionObj outerGroup, ProcessingContext processingContext, MatrixInstance matrixInstance, Boolean outerGroupings, RuntimePivotGroupRootObj currOuterHeadingGroupRoot, MatrixHeadingInstanceList headingInstances, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeMatrixObj.CreateInstances(ReportItemInstance riInstance, IList instanceList, RenderingPagesRangesList pagesList)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeRICollection.CreateInstance(ReportItem reportItem, Boolean setupEnvironment, Int32 index)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RuntimeRICollection.CreateInstances(ReportItemColInstance collectionInstance, ReportItemCollection reportItemsDef)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.Merge.CreateInstances(ParameterInfoCollection parameters, Boolean noRows)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.Merge.Process(ParameterInfoCollection parameters, Boolean mergeTran)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.ProcessReport(Report report, ProcessingContext pc, ProcessingContext context)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.ProcessReport(Report report, ProcessingContext pc, Boolean snapshotProcessing, Boolean processWithCachedData, GetReportChunk getChunkCallback, ErrorContext errorContext, DateTime executionTime, CreateReportChunk cacheDataCallback, ProcessingContext& context)
at Microsoft.ReportingServices.ReportProcessing.ReportProcessing.RenderReport(IRenderingExtension renderer, DateTime executionTimeStamp, GetReportChunk getCompiledDefinitionCallback, ProcessingContext pc, RenderingContext rc, CreateReportChunk cacheDataCallback, Boolean& dataCached)
--- End of inner exception stack trace ---
w3wp!webserver!1!10/25/2006-16:52:41:: e ERROR: Reporting Services error Microsoft.ReportingServices.Diagnostics.Utilities.RSException: An unexpected error occurred in Report Processing. ---> Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: An unexpected error occurred in Report Processing. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
w3wp!library!1!10/25/2006-16:52:41:: e ERROR: Found System.OutOfMemoryException exception: Microsoft.ReportingServices.Diagnostics.Utilities.RSException: An unexpected error occurred in Report Processing. ---> Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: An unexpected error occurred in Report Processing. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
w3wp!library!1!10/25/2006-16:52:41:: e ERROR: Terminating worker process
Any suggestions?
Thanks,
Andrew
October 26, 2006 at 11:26 am
I don't want to sound like a pain but here's a few thoughts I had on this last night...
Is your test report truly indicative of the reports you're looking to run, or is it more a case of can i build something that can make RS fall over? While I personally agree that maybe it should be able to handle this, building things to break things isn't that hard to do (even with the old Foodmart cube you could quite easily run an MDX statement that would make the CPU sit at 100% for hours if not days).
If the report is a valid example fo what you'll be doing, then is the grouping necessary in the report? ie could you group in the sql statement and let the relational engine to the heavy lifting?
If the grouping is required in the report, possibly you could add an additional filter to reduce your rows returned. The way I figure it, you've either got a small number of groups each with a very large number of rows, or a large number of groups with a small number of rows grouped beneath. In the former, i would argue that you should add a filter to the report as anyone opening a group will be hit with (probably) many thousands of child rows. In the latter, you'd be presented with a large nunber (100's, maybe 000's) of groups which again could be overwhelming and difficult to find what you're actually looking for within the report.
Assuming that all of the above is not true and you do really (truly ruly) want this report, I would start by debugging the sql used to retrieve the data -> see how long it takes to return a result when just running the sql in sql mgmt studio. This is your baseline (ie RS can't do it any faster unless it's cached) for raw unformated data time. I'd also look to the client stats to see what volume of data 180MM rows actually is (ie in bytes/kbytes). If the total is > 2gb (or whatever the threshold is set to) this could be why it can't do the grouping. To an extent, regardless of how long this takes, you're likely to find that your answer is 'get more ram'. re whre you've indicated that it's not going to the thresholds you've set, which memory usage stat are you reviewing? It could be that you're looking at how much physical mem the process is using and i thouht at least one of those settings you mentioned referred to total memory (physical and virtual) being used. Is it possible that the total memory usage of the process broke the threshold which then made the service recycle?
Let us know how you go/what path you take. As I said, personally I'd like to think with 2gb of mem that RS could do that grouping, but i guess it depends on how much memory each row takes.
Steve.
October 26, 2006 at 10:22 pm
Hi Steve
Thanks for taking an interst in this.
The rows in the resultset are not very wide - 20 columns that total, at most, 376 bytes each. That's a maximum total of about 65mb accross the whole resultset, and, according to the client statistics output in Management Studio, it's only 41.5 mb. Four instances of that aren't going to make much of a dent in two gigs.
This is a report that is currently used in production (via a home-grown reporting package), so the grouping, volume of data, etc, is what is required. However the report is too large to run without filters in our current reporting package. We're looking at migrating to Reporting Services for this (and other) reasons, so we're testing how well Reporting Services will handle our biggest, nastiest report.
My personal opinion is that a report of this size couldn't possibly be human-readable, and should be broken down into a parent report and sub-reports. However, I'm told that isn't an option. Besides, we are trying to see what it will handle.
The memory statistic I've been monitoring is just the "Mem Usage" column in Windows Task Manager.
I have made a breakthrough this afternoon however. I tried tweaking the properties of the ReportServer Application Pool in IIS, and have found that increasing the Maximum Number of Worker Processes helps - all four reports finished in under eight minutes. Balancing the work load between these processes seems to be ensuring none of them use too much RAM.
I'll need to do some more investigation in this direction, I think.
Thanks,
Andrew
October 29, 2006 at 3:53 pm
That's great, thanks Steve - both good articles.
Thanks,
Andrew
Viewing 9 posts - 1 through 8 (of 8 total)
You must be logged in to reply to this topic. Login to reply