What Should Be In a Review?
Introduction
I wrote an article recently about how we review products at SQL Server Central.com and how we might make things better. I received a number of suggestions and appreciate them all. These have spurred some debate at SQL Server Central as well as given us some items to discuss with vendors.
Given that feedback (and I encourage more from others out there), I thought the next logical step was to propose and layout a format for reviews and get some feedback on that. I know that a number of you had some good ideas and I incorporated some of them into this proposal. Please read it and think about it and give your feedback. Even it's as simple as a "like it" or "hate it", take a minute to write a response. It really helps us and the community and it will hopefully allow us and the members to provide more consistent information and feedback to all of you.
Lastly, tell us how you heard about this product. Recommendation, vendor email, etc.
Review Format
Brian Knight has a pretty good template from which I'm starting. It includes a number of items that I like, though I expanded it a little and added come comments on each section. This is what I would propose:
| Section | Description | Metrics |
| Introduction | This should include the name of the product and a basic description of what the product should do, what problems it solves, or what functions it enhances. It should also include the version of the product and whether this is beta, rtm, or patched software. | Version of software |
| Environment | A description of the environment in which you expect to use this software. This should include hardware and software descriptions, but also the network and operating (business) environment in which you are looking to deploy the software. If this is for your job, then it should be a description of the size or your environment and in what place you expect this software to fit in. This is the place where you lay out your expectations, which should allow other readers to get a sense of what you were looking for. It may help those of us with much different environments to not be turned off if the software doesn't fit you. It might fit us instead. An example of this for my work would be: I work in a large company, 8000 nodes, 60+ SQL Servers, and expect this software to be deployed to consolidate the error logs on all my SQL Servers into a central location and parse them for errors meeting keywords that I have entered. I would probably write this differently, but that is the idea. | Hardware used or required, CPU, RAM, disk, etc required. Include the test machine(s) config. |
| Installation | Walk the user through the installation or upgrade of the product. Use a colloquial style, letting us know what issues you had, things you did wrong that you could have avoided by reading the manual (who's had that happen?), etc. Your general impression of setting up the software. Images can be included (and are encouraged). | Time to install, including mistakes. |
| Using the Software | A description of the way you used the software. Walk the user through how you started using the software, again, including any mistakes you made along the way. If you made them, then it's likely that someone else will make them as well. It's nice to see images here that describe what you are writing about for the user. For many readers, they will likely have never seen this software. You should describe the features that you used and how they reacted to your testing. You should also describe the features you know about, but don't need or don't test. These should be objective observations of what has occurred as you worked with the product. | none |
| Technical Support | Details of any support you received. If you didn't need to call, let us know. | none |
| First Impressions | This is the place to give your opinion. Let us know what your first impressions of this software were, what you liked, what you didn't, bugs, etc. By limiting opinions to this area, we know that we should view this as an opinion as opposed to fact. We should also be aware of the environment in which you are working from the Environment section above. This isn't a license to slam a product or vent, but rather a place where you let us know things that you didn't like or liked. Give us an opinion, but be professional and respectful. If you had issues with technical support, also include them here. | none |
| Conclusions | Give us a couple final notes here. Overall usability, did it solve your problem, would you or did you purchase it. Is it deployed? You don't need to rehash what you wrote above, hopefully we've read that, but this is the place to let us know if you're recommend this product or if you would purchase it. I know there are product you'd like, but don't have budget, so let us know if you could get approval, would you buy it. Also tell us (And the vendor) if you feel it's a good value for the money. | Price/Value. Buy decision and/or recommendation. |
| Ratings | A rating of various categories, which are described below. | none |
| Product Information | This is the place where you include the contact information and pricing information for the product. These should be published facts from the vendor. If possible, contact them and get this directly from them. | Vendor name, address, phone, email, etc. contact information Product name and version Pricing for products, include quantities or CPU numbers with prices, or provide the contact for pricing questions. Also include if demo or evals are available. |
Whew that's a lot. Certainly adds some work to my review process, but I think it might help get some consistent reviews. Now for the ratings.
I tried to pick a number of categories that would apply to all products and would cover a number of items that I want to know about a product. I also included the feedback I've gotten from people on reviews. All items should be rated as:
- 0 - Not applicable. Not used in an average.
- 1 - Bad, not recommended, didn't work, fatal error, something that turns me off or causes the product not to work.
- 2 - poor. Not meeting expectations, not working, not critical.
- 3 - Average, didn't impact me, didn't' affect operation, nothing special
- 4 - Better than average, works well, does what is advertised, something I liked
- 5 - Excellent, exceeded expectations, enhanced the product, made me want to buy/recommend this product.
I know the categories don't match each other perfectly, but I'm trying to get a feel for where you stand or where the product stands in various areas. A single sentence can accompany each rating for clarification. The total rating for the product is based on an average of each area. 0s are not counted in the average.
Ratings
Ease of Use
How easy did you find this software to use? Keep in mind the complexity of the software and it's relative ease to similar products, including any native SQL Server tools.
Guideline
- 1 - Unusable, difficult and cumbersome. Not intuitive.
- 2 - Hard to use or understand. You got it to work, but struggled.
- 3 - Average compared with other software, needed some instructions.
- 4 - Easier to use, able to figure things out without resorting to the manual.
- 5 - Easy and intuitive. You enjoyed this software.
Feature Set
What do you think about the feature set? Are they as advertised or do you feel that there are features lacking that should be in the product. Note these items in your review and rate it accordingly here.
Guideline
- 1 - Missing features advertised, missing features that you feel should be there, features not working.
- 2 - Missing some features or they do not work as advertised.
- 3 - Most features in product and work
- 4 - All features work
- 5 - Contains enhancements that surprised you or were not advertised.
Lack of Bugs
Is this software bug free? Probably not, but if you didn't find any, then rate this a 5. I'd degrade the software from there depending on how the bugs affected the operation of the program or the stability of the test system.
Guideline
- 1 - Bug ridden, unusable
- 2 - Operation of the program was affected or stability affected.
- 3 - System stable, but some bugs exist with no workaround.
- 4 - Might be some bugs that are either annoying or a workaround exists. Shouldn't affect program operation.
- 5 - No bugs found.
Value
This is a tough one because everyone has different sizes and parameters for budgets, but imagine you run a dot com on a shoestring. Is there value in spending your hard earned money on a product like this? Is it saving time and/or money in relation to it's cost. 5 is a great value, 1 not likely or no.
Guideline
- 1 - Not worth spending money on this product, you would not be likely to purchase this product.
- 2 - You probably wouldn't purchase this product for your environment, but you wouldn't rule it out.
- 3 - Not sure where this stands as a value proposition.
- 4 - A good value and you're inclined to purchase this.
- 5 - You would purchase this product as it's a great value.
Technical Support
If you didn't contact support, rate this 0. Otherwise, how did it go in your opinion. Take response time, completeness of response, accuracy, etc. into account. The details of your interaction should be above.
Guideline
- 1 - Non existent, no response, rude.
- 2 - Poor or slow response, did not solve issue.
- 3 - Solved issue, but slow or poor response.
- 4 - Solved issue, responsive in a timely manner.
- 5 - Exceeded expectations.
Documentation
How good are the docs? Are they accurate? Do they cover the features?
Guideline
- 1 - Incorrect and/or incomplete. Not helpful.
- 2 - Errors or incomplete, but some help.
- 3 - No better or worse than other software.
- 4 - Good documentation, few errors, easy to read.
- 5 - Covers 90%+ of what you need and is complete.
Performance
How the software runs? Does it work well, not eat up resources, meet the expectations in your environment. And it solves the problem or provides the enhancements you wanted.
Guideline
- 1 - Doesn't run on the test system, doesn't work.
- 2 - Doesn't solve problems or significant impact.
- 3 - Program runs ok, impacts system, still solves problems.
- 4 - Program runs well, minimal system impact, solves most problems.
- 5 - Program runs great, no impact on system, solves problems.
Installation
What is the installation like? Does it work well? Confusing? Any issues?
Guideline
- 1 - Installation didn't work. Or significant workaround outside of the installation program.
- 2 - Problematic installation, slow, confusing.
- 3 - Average installation, requires some paying attention.
- 4 - Installation easy, worked well, made sense.
- 5 - A breeze, almost unnoticeable.
Learning Curve
Guideline
- 1 - You couldn't learn to use this software and suspect it would be very difficult to learn.
- 2 - You had trouble learning to use this software, requires significant time to learn to use.
- 3 - Required reading some documentation, but able to learn in a reasonable amount of time.
- 4 - Pretty easy, but had to fumble around to figure it out.
- 5 - No curve, this was intuitive and easy to use.
Conclusions
That's my proposal for a review standard. I'd like to see and hear comments, criticisms, suggestions, etc. from those of you that work with SQL Server and like to get information about tools. We would like to provide that for you, but we want to be sure that we are providing you with something valuable.
Please take a moment and submit some comments about this proposed format and I'll update it in a couple weeks based on your comments.
Steve Jones
©dkRanch.net April 2003