Large Chunks of Data

  • @AndrewJacksonZA - the Rick Astley pie chart made me laugh out loud....the 'glider' avatar indicates you are obviously a Conway's Game of Life fan too 🙂

  • We found a "unique" way to resolve all of this at a previous company I worked for...

    ... The users wanted all the data...

    ... so we gave it to them!!!

    We gave the users their own "report" server and we updated it every night at midnight with one of those very nice EMC clone/snapshot updates. And, believe it or not, it worked out very, very well. No universes to maintain, no huge datamarts to build, no special requests to handle, no report maintenance. The users (all in house) did it all (most of the authorized users had SA privs and ... what are they gonna do? Blow up a copy that we could have back on line in 10 minutes?) and they were tickled to death to do it. Us too! About the only thing we had to do was answer the occasional "so how do I...". It gave us lots of extra time to make sure the production data that we'd fire off at the report server every night was actually correct. If someone crushed the report server with a bad query, we'd just kill the spid and call that person up. If they did it twice in the same day, a couple of us would all pay that person a personal visit. It didn't take people long to learn about such things. Again, I say, it all worked out very, very well and the extra server was worth it's weight in gold in IT hours saved and ultimate (in-house) customer satisfaction! 😉

    Heh... If you decide to do the same thing, do remember to NOT make the server public facing. 😉

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • @steve-2,

    As usual, good and timely topic with lots of good discussion that follows. Thanks for all that you do. 🙂

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • Jeff Moden (9/19/2010)


    We gave the users their own "report" server and we updated it every night at midnight with one of those very nice EMC clone/snapshot updates. And, believe it or not, it worked out very, very well.

    I agree Jeff, if your users are at all technically inclined this does work out nicely. The initial ramp up is *painful* until you get your users talking to each other for the 'how do I...' stuff, especially when the new guy comes on. Usually takes about 2 months and then you're home free... mostly.

    However, if you're dealing with the sales staff of doom who are more used car salesmen than slightly geek inclined (think that old guy with the cell phone you can use as a doorstop)... this sadly doesn't work out so well. 🙂


    - Craig Farrell

    Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.

    For better assistance in answering your questions[/url] | Forum Netiquette
    For index/tuning help, follow these directions.[/url] |Tally Tables[/url]

    Twitter: @AnyWayDBA

  • Craig Farrell (9/19/2010)


    ...think that old guy with the cell phone you can use as a doorstop... 🙂

    Hey now! Watch it! I strongly resemble that remark! 😛

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • Whenever someone requests the creation of a new report, and the specifications don't seem clear or feasible (for example 50 columns fit upon an 11' landscape page), then I'll ask the business analyst or client to provide a mockup version of the report in Excel. If they can't logically visualize what the end result should look like, then they can't judge the end result of the development effort. What typically happens is that they meet behind closed doors for a couple more days and then come back with a revised set of requirements.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • Eric Russell 13013 (9/20/2010)


    Whenever someone requests the creation of a new report, and the specifications don't seem clear or feasible (for example 50 columns fit upon an 11' landscape page), then I'll ask the business analyst or client to provide a mockup version of the report in Excel. If they can't logically visualize what the end result should look like, then they can't judge the end result of the development effort. What typically happens is that they meet behind closed doors for a couple more days and then come back with a revised set of requirements.

    That's a good idea. Get them to think about it and understand when their requirements aren't making sense.

  • Steve Jones - SSC Editor (9/20/2010)


    Eric Russell 13013 (9/20/2010)


    Whenever someone requests the creation of a new report, and the specifications don't seem clear or feasible (for example 50 columns fit upon an 11' landscape page), then I'll ask the business analyst or client to provide a mockup version of the report in Excel. If they can't logically visualize what the end result should look like, then they can't judge the end result of the development effort. What typically happens is that they meet behind closed doors for a couple more days and then come back with a revised set of requirements.

    That's a good idea. Get them to think about it and understand when their requirements aren't making sense.

    Heh... I tell the user that they're going to need a printer that can print on Tabloid (11"x17") paper or that we'll need to print vertically on green-bar. 😛 And, yeah... I've actually done the greed-bar thing to produce a report that was some 50 pages wide on a "single sheet". It was... ummmm... ridiculous. 😛

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • Jeff Moden (10/21/2010)


    Steve Jones - SSC Editor (9/20/2010)


    Eric Russell 13013 (9/20/2010)


    Whenever someone requests the creation of a new report, and the specifications don't seem clear or feasible (for example 50 columns fit upon an 11' landscape page), then I'll ask the business analyst or client to provide a mockup version of the report in Excel. If they can't logically visualize what the end result should look like, then they can't judge the end result of the development effort. What typically happens is that they meet behind closed doors for a couple more days and then come back with a revised set of requirements.

    That's a good idea. Get them to think about it and understand when their requirements aren't making sense.

    Heh... I tell the user that they're going to need a printer that can print on Tabloid (11"x17") paper or that we'll need to print vertically on green-bar. 😛 And, yeah... I've actually done the greed-bar thing to produce a report that was some 50 pages wide on a "single sheet". It was... ummmm... ridiculous. 😛

    This situation was more of an issue, say ten years ago. Fortunately today we have better BI reporting tools that allow the users to easily create their own reports from a reporting model or staging table containing the requested 50 columns. Business can tinker around with the process of making it all fit on a single page themselves or offload the task to a hapless intern.

    😉

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • Ah, its refreshing to know that I'm not alone in my quest to quell the uprising of 1,000 page reports. "A data dump is different than a report. SSRS is for reports. If you want a data dump, let me set up an export job that will dump to the filesystem for you."

  • Here in the US, SOX regulations require certain types of companies retain printed hard copies of critical financial and operations related reports, and this the source for a lot of these 1,000 page / 100 column requests.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • The total number of pages on a report a user wants to scan through hasn't changed in five years. However, there are scenarios where an oranization prints 1,000 page reports to paper, not because someone will actually read it today or ever, but because they want to immediately file them away for safe keeping.

    Sometimes executive managment can rest easier knowing that in the aftermath of a zombie apocalypse, or in the unlikely event that this computer revolution thing turns out to be a passing fad, they actually have printed copies of all their financial reports to fall back on. Sometimes it's just mandated by a government regulation or a business policy handed down since 1920.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • Eric,

    Are they just afraid of electronic signatures?

  • At my previous employer I wrote reports for a government agency that contracted with us. They requested to have historical data included in the report even though that data never changed. At first it was fine because they were new and there was not too much data, as time passed it broke the export to excel limitation(SQL 2008 R2) but they basically refused to accept anything but the full report. I always question why they needed all of that old information since most of it was out of date and new processes and been implemented, and why they wanted so much of it. even 65,000 rows of data I think is too much for any one to really make much sense of.

  • Timely topic. After several years of holding an BI and SSRS environment together on an aging and storage limited server, we were told to create a "Next Generation BI" environment. Good to get funding and go from 1TB to 5 TB, difficult to get directions/requirements/and tools. This is all SQL Server 2014 in a Microsoft shop an expectation of new Data visualization.

    Lots of churn. Everyone wants to use the words of the day: "Big Data", "Predictive Analytics", "Dynamic Visualization", "Data realm expansion", "User Driven Reporting" ... The starting point is obvious, replace the DB server and make sure existing ETL and SSRS are stable. Picking a path with limited resources has been difficult. The ETL wish list could burn 100% of resources, and management expects frequent "Sales presentations".

    Leads to questions:

    1) How to collect rational "use cases", and sell the product?

    2) How to translate Microsoft Excel 2013 Power View into customer deliverables with/without SharePoint 2013?

    3) How to Improve ETL deliverables (Avoid the costs of SSIS package development)?

    4) How to create simple Dashboards while allowing drilldown to the underlying 1 TB of detail.

    5) Creating a Demo project that does shows "Big Data" concepts at minimal cost.

    6) Making this fit into an MS Project schedule with milestones and EVM.

    Everyday provides a new set of directions and questions. Wondering how it's done in the larger world!

Viewing 15 posts - 31 through 45 (of 49 total)

You must be logged in to reply to this topic. Login to reply