How Do You Review a Product?
Introduction
We here at SQL Server Central.com appreciate your participation and patronage at our site. And we hope that we are providing a valuable and useful community for the SQL Server users out here. Short plug: tell your friends and colleagues about us and please send us feedback.
Enough of that. I titled this article the way I did because we are facing a small quandary at SQL Server Central.com. We are being asked to do an increasing number of reviews for products by vendors (all different vendors). The workload is high, but that is not the issue. We've been struggling with finding a review process that is fair and useful and one that is desired by the community.
Confused? Well, so am I a little. Let me try and explain the review process as it exists and what we try to achieve and then give you some comments and ideas we have. I'm hoping to get feedback from you readers that will help us and the vendors continue to move forward.
Obtaining a Review
We have a policy posted here that outlines how one can request a review. It's a general, send it to us and we'll take a look, we're busy so it may take awhile, etc., etc. It 's a general policy that may be outdated, hence I'm writing this.
When we receive a query, the three of us (Andy, Brian, and myself, will take a look and try to determine if one or more of us has the time to do the review. If so, we will then work with the vendor to get a copy of the software. We note on the review page that we want a non-expiring copy, mainly because we may take longer than the 30 days most versions give you, but we haven't been completely unforgiving on that point.
At that point, someone will install the software, at home, at this site, our dev site, somewhere that it can be evaluated as to it's usefulness, ergonomics, usability, etc. This can be a time consuming process, especially if there are features that are not intuitive or bugs exist. Just like any other user, we may contact the vendor's tech support group to get assistance. Whether the techs know we're doing a review or not is not something I can say, but for the most part vendors are eager to please all potential customers, so we tend to get support.
Now the fun part. Once one of us spent some time with the product, we write the review. We all have slightly different styles, but we try to cover the installation, what we liked, what we didn't, what worked, what was broken or incorrectly explained, and what support we received from the vendor. We also include the specifics you need to purchase the product and assign a rating, 1-5 stars.
Now the part you may not be aware of. We usually submit this review to the vendor. Not for them to change, but for them to correct technical inaccuracies, like us misstating the behavior of a feature, or perhaps saying that there is no debug option when there is. The vendor can give us feedback about these items, including things we may have misworded. Branding is important to everyone selling a product, including us here, so if we've called something a "product", but the vendor prefers we call it a "software utility", we tend to listen and alter the words to align them with the vendor.
You may be getting concerned here about our reviews and their impartiality, but let me be clear. We do not allow the vendor to rewrite the review. They can make suggestions or comments, but the final copy is determined and approved by us.
The Sticky Part
What's sticky about that? Well, think about bad reviews. Sometimes you've gotten a product and it didn't work. Bugs, issues, whatever, it was way more trouble than it's worth. Or you didn't find it useful. Or perhaps it didn't live up to claims. Or maybe you just didn't like it. I've run into all these items and written them.
And sent my review with these remarks to the vendor.
Obviously the vendor isn't happy. They may make comments or want things changed. They may say that the "next version" will include some of these things. I'm happy to include that comment, but I don't change the review to remove things that I have observed, good or bad.
So what do I do? Stop for a minute and think about something. SQL Server Central.com is a fairly large community at this time. We average over 900kb/s of bandwidth for the site. That's AVERAGE, not peak. Running a site that basically requires a T-1 isn't cheap. In fact, it's a substantial amount of fixed $$ every month. Plus we pay for content (not much, sorry 🙁 ), and we upgrade equipment. In fact we just bought a new server that should be live by the time you read this. We don't expect to make a bunch of money from this, but we can't support it out of pocket either.
So we depend on advertising mostly for now. We're trying to find other revenue streams, like our Best Of Book project. Our main focus is to continue to server the community without resorting to a paid subscription model, something none of us likes.
That puts us in a tricky position with vendors. On one hand we depend on them (for now) for support and we don't want to do something that jeopardizes their business or hurts sales. On the other hand we don't want to misreport things to the SQL community. We use SQL Server and various third party utilities as well and often look to others for opinions and reviews before we purchase something ourselves.
To date, we've kind of walked a fine line by allowing a vendor to request that we don't run the review and for the most part, we've just not published our thoughts, preferring to say nothing than say something bad. However, before you start complaining and saying that we're doing a disservice by not warning you about poorly written software, consider a couple things.
My opinion of a software product is based on my own experiences with the product, but also on how it worked in my environment. The SQL Server Central environment is different than my work environment. What works in one place may not work in the other. My workload also affects my opinion. I may be busy and miss things or not feel it's worth the effort learning some new product, something that you may really need to concentrate on to get a fair review. Or get a good opinion of how the software may really help you.
There's also another thing we've learned over the last two years. Sometimes a software product works better for someone else even though you don't see the benefit. Andy and I had a product like this. He kept telling me how great it was, how it saved him money and time, and paid for itself in weeks. I looked at it, but it honestly didn't work for me. Not that it didn't work, but it wouldn't save me a lot of time because I didn't have the need for that functionality very often and probably wouldn't have written a great review.
We've debated about this point. We suspect that there are lots of software products, perhaps some that we've reviewed (published or not), that fall into this same situation. It doesn't work for me, but it might for someone else. In this case, we tend to feel that publishing a bad review isn't in the best interests of anyone. Not the vendor nor the reader. Obviously it can hurt vendor sales, but the reader may get a bad impression and not try it, when it might really be a benefit to him or her.
Help
We'd like to hear some opinions and suggestions from you about what you like to see in a review and how they should be conducted. Keep in mind a few things.
- We want SSC to continue to be free
- We want to provide information to the community about software available.
- We want to be fair to everyone
We've had some suggestions from vendors and we've brainstormed a little. Some things we've considered:
- Paid reviews - Not that this makes us impartial, but it's something we've considered. Having the vendors pay us if they want a review. Not sure this model works.
- Informational Reviews - Some other sites do reviews of products that don't really provide an opinion, just give the facts about the product. Something user written, but more like a press release or brochure. To be honest, I don't like this type of review and am not sure of it's value, but perhaps people would like to see information about a product that doesn't come from the vendor and doesn't contain an opinion.
- Brutally Honest Reviews - A vendor provides us with a product and we publish whatever we want. This is our current model, though we are sensitive to hammering a product. If this happens too often, we lose advertising, lose revenue, and SSC goes away. Like many sites have already gone.
Conclusions
We'd like you to think about it and give some feedback. Not so much about what our model should be, but more about what you expect from a review and/or what you'd like to get out of it. We have built a great community with everyone's help and we want it to continue to grow.
Steve Jones
©dkRanch.net March 2003