Sounds like you're removing your detail group i.e. your lowest grouping level that tell the report what field to iterate over to display. Without this piece of info it's aggregating over your whole data set.
If you're looking to return only some rows that are more relevant than I would modify your query or add filters on the dataset.
If you're looking to just cut down on the number of rows then using a 'chunkier' group (e.g. state not town) or by filtering to show the top N rows.
If none of this helps, holler 🙂